Introduction to the Analy i 


Competing Hypotheses (ACH) 


To start the module, click the Next button. 


No audio 
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Learning Objectives 


= 
(e Learning Objectives 


By the end of this session, 
you will be able to: 


Create an ACH matrix 

Explain the benefits of using ACH 

Fully appreciate the value of diagnostic evidence 
Use ACH appropriately, fully aware of its 
weaknesses and limitations and 

Recall the Eight Steps of the ACH process. 


Hello, my name is . , am aftanalytic meéthodologist. | study how analysts 
solve tough problems. 


By the end of this session, you will be able to: 


° Create an ACH matrix, 

° Explain the benefits of using ACH, 

° Fully appreciate the value of diagnostic evidence, 

* Use ACH appropriately, fully aware of its weaknesses and limitations and 
e Recall the Eight Steps of the ACH process. 
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ACH is a SAT 


ACH is a Structured Analytic 
Technique (SATs). 


Í Did You Know? 


SATs are analytic approaches that have been 
formalized (usually into checklists). 


ACH is taught to all analysts because: 


It is versatile (for tactical and strategic problems) 
It can be done be one person or a group; and 


It can be done quickly on the back of an envelope, 
more formally over a long period or with software. 


ACH is a Structured Analytic Technique (or SAT). 


SATs are analytic approaches that have been formalized (usually into checklists). The 
term was coined by Kathy Pherson, a former CIA anayst and wife of author Randy 
Pherson who popularized the term in his 2011 book "Structured Analytic Techniques 
for Intelligence Analysts." 


ACH is a fundamental technique, taught to all analysts because: 


° It is versatile (good for both tactical and strategic problems) 
* It can be done be one person or a group; and 
° Gan ‘bedone on the back of an envelope or more formally over a longer period 


with software, ^. 
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Setting up an ACH Matrix 


Evidence 
Evidence 
Evidence 


Evidence 


Analysis of Competing Hypotheses (ACH) is a Simple model for how to think 
about a complex problem when the available information is incomplete or 
ambiguous, as typically happens in intelligence analysis. 


It is a process for making a well-reasoned, analytical judgment that begins by 
arraying evidence against hypothesis. 


It is particularly useful for issues that require careful weighing of alternative 
explanations of what has happened, is happening, or is likely to happen in the 
future. 
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Rating evidence 


Each piece of evidence is rated against each hypothesis 
as Consistent, Inconsistent, or NA. 


Evidence 
Evidence 
Evidence 


Evidence 


With the evidence iri rows and the hypotheses in,columns, the next step is to 
rate each piece of eviderice/agádinst.each/hypotheses as either consistent, 
Inconsistent, or Non-applicable. 


This helps the analyst overcome, or at least minimize; some of the cognitive 
limitations that make prescient intelligence analysis so difficult. 


It is important at this stage to work horizontally, not vertically. This way, each 
piece of evidence is re-examined in light of a different hypothesis. 


ACH is grounded in basic insights from cognitive psychology, decision analysis, 
and the scientific method. It helps analysts protect themselves from avoidable 


error, and improves their chances of making a correct judgment. 
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Disconfirmin 


Evidence which is inconsistent helps to disconfirm a 
hypothesis. 


Evidence 
Evidence 
Evidence 


Evidence 


Although the naturat tendéncy is to focus on the hypothesis with the most 
"C's" the “l's” are more importarit because they help to disconfirm a 
hypothesis. This is important becaüse whilé additional evidence can usually be 
found to help “prove” a pet theory, this evidence ónly advances us 
incrementally to certainty. We might, for example, hàve enough evidence to 
be 85% certain, then find something that allows us to be 90% certain. This 
game can continue indefinitely. 


Our hypothesis could be, for example, that all swans are white. This is based 
on the fact that we've observed 100 swans, all of whom were white. We could 


^; try to confirm our theory that all swans are white with some research. Every 


time we find a.new white swan, we feel more certain that our hypothesis is 
correct: n | 


OU Da “However, iall that is Héeded'to.blow our hypothesis out of the water is one 
black swan.This.one piece of inconsistent evidence disconfirms our 


hypothesis; ; 7^ 
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ACH: Definition 


ACH involves: 


the identification of a complete set of alternative A S, 
hypotheses, | 


the systematic evaluation of data that is consistent 


and inconsistent with each hypothesis, and 


the rejection of hypotheses that contain too much 
inconsistent data. 


ACH involves the identification of'a.completé Set of alternative hypotheses, 
the systematic evaluation of data’that iS.consistent and inconsistent with each 
hypothesis, and the rejection of hypothéses that contain too much 
inconsistent data. : 
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ACH: Why Should I Use It? 


ACH helps overcome common analytic pitfalls: 


e Rushing to a preferred explanation or-canclusion 


* Ignoring or discounting information that doesn't 
"fit" the preferred explanation. 


* Failure to consider a full set of explanations or 
hypotheses. 


These are examples of the need for closure, 
confirmation bias, and tunnel vision. 


Most analysts already‘use ACH. without knowing it. They look for the explanation that 
best fits the available evidence. However,‘there is great value in formalizing the 
process. 


Done properly, ACH helps overcome some common analytic pitfalls. There is, for 
example, the tendency to rush to a preferred explanation. "Who's the bad guy? Oh, 
its probably the guy with the beard!" Once an opinion or theory is formed, we have a 
tendency to ignore information that doesn't fit. False confidence in our pet theory 
contributes to the failure to consider a full set of possible explanations. These are 
examples of the need for closure, confirmation bias, and tunnel vision. 


Some people have a stronger need for closure than others. Analysts must be able to 
tolerate uncertainty and juggle more hypotheses for longer periods than most 


Y. ipgople. 


The: confirmation bias is the natural tendency to only look for evidence that 


> confirms’ our; ^hypethesis and not evidence that discounts or disconfirms it. This 
prang i is hardwired Iñto our brains. 


Ketording to thé Depa rtment » Justice, tunnel vision is the leading cause of wrongful 
convictions in, Carada and-elsewhere. Source: Working Group on the Prevention of 
Miscarriage's of Justice, ‘September 2004. 
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ACH: Why Should I Use It? 


ACH is a useful method to systematically review and 
evaluate information. 


* Helps identify most diagnostic information s 


* Allows you and other analysts to identify areas of) A. 
agreement and disagreement 


* Useful for considering the possibility of Denial 
and Deception 


As the complexity of-a problem incréases, the néedfor a systematic approach grows. 


Many analysts find ACH useful as a data management tool. Ona single page, you 
have everything pertinent to the problem, all nicely organizéd. This makes it easy to 
review and evaluate information, alone or in groups. 


With free software like the ACH Tool, it is easy to sort the evidence by diagnosticity, 
allowing you to focus on key pieces of evidence. 


ACH also provides a great framework for group discussion. Working through an ACH 
Matrix with a colleague or group, ensures that nothing is missed. It also exposes 


where there is disagreement. 


Finally, by highlighting inconsistencies, ACH can also expose when an adversary is 


A attempting to fool us-by planting evidence that leads to a false conclusion. The 
bš Ceyidence, for example, could be Consistent with a military exercise, but if a foreign 
~~ “leader has beén taken to á sécuredcation, it is possible that a real attack is 


imminent, “E/G, ` 
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Diagnostic evidence or information, 
helps you to assess which hypotheses’ | 
is likely to be correct. 


Important 
Evidence that is consistent with all: 
hypotheses, has no diagnostic value. 


C C C C-not diagnostic 
| | C | -highly diagnostic 


It is important to understand the Keep! of “didgnosticity’. 


Generally speaking, evidence that | is consistent with all hypotheses, has no diagnostic 
value. 


A high-temperature reading may have great value in telling a doctor that a patient is 
sick, but relatively little value in determining which illness a person is suffering from. 
Because a high temperature is consistent with so many possible hypotheses about a 
patient’s illness, this evidence has limited diagnostic value in determining which 
illness (hypothesis) is the more likely one. 
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S 


5 D One ies M L I 
Guess which one of the dallowdns Palo perd à Any 47, d A Os 
When someone is lying, they: S r=) On, 


Touch their nose Q 
© Stutter fA 
YO 
© Switch to the passive voice 2 WE; Š 


© Avoid eye contact 
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Wusio Paying 


Diagnostic evidence orinformation is’ 
the focus of your argumentit is the 
reason you reach a particular, * 
conclusion. 


Ensure it is accurate. Is the source of 
the information reliable? 


Consider alternative explanations or 
interpretations. This often generates 
new hypotheses. 


Diagnostic evidence or information is the focus of your argument. It is the reason you 
reach a particular conclusion. 


With ACH, a common experience is to discover that most of the evidence supporting 
what you believe is the most likely hypothesis really is'hot very helpful, because that 
same evidence is also consistent with other hypotheses (i.e. all'C’s across the board). 


Once evidence is identified as highly diagnostic, it merits close examination. After all, 
it is likely the reason that you have reached a particular conclusion. 


First, ensure it is accurate. Where did this information come from? Who collected it? 
When atid NOW? Is the source of the information reliable? 


re Second, ask yourself. Pag evidence could be interpreted in a different way. In the 

MAD classic movie, ‘Twelve Angry Men” the jury is ready to convict, but for the efforts of 
the fone holdout who begins to éystematically challenge the evidence. An alternative 
explanation for the evidence often leads to a new hypothesis. 


12 
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ACH: When Should | Use It? 


° When you have a large amount of data to absorb; < 
and evaluate. Msa Vd i 


Hypotheses 


Consider using ACÁ whenever yoü.have a large amount of data to absorb and 
evaluate. 


13 
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ACH: When Should | Use It? 


When you need to be thorough. 


When you want to challenge your (and others’ YA 
evaluation of the evidence. 


When you need to carefully weigh alternative 
explanations of what has happened, is happening, 
or is likely to happen. 


When you need to keep a record 
of the process. 


There are four other good reasons for ifivesting time“and effort in a formal ACH process. 


It is a good argument to say that you Raye evidenceto support your conclusion. It is a better 
argument to say that you have also considéred and. discounted.alternative explanations. “I 
believe it is “X' although it might be ‘Y’ or even possibly ‘Z’ /but it-is definitely not ‘A’, “B”, or 
'C'. ACH contributes to more thorough analysis. ; 


Everyone has a personal “lens” thorough which they view the world. When a familiar situation 
arises, past experiences provide us with the ability to respond quickly. On occasion, however, 
we might be led astray. ACH provides a means whereby others can follow our reasoning and 
challenge our evaluation of the evidence. 


In our personal lives, the consequences of poor analysis are usually not dire. In our 
professional lives, poor analysis could lead to loss of life. This necessitates carefully weighing 


/;,. alternative explanations of what has already happened, is currently happening, or what is 
likely, to ie 


It is SIKA in the nature of our work that it will be audited to ensure it complies with policy. 


i ‘Complaints willbe investigated by, oversight bodies, and, when failures occur, there will be 
CGommissions of Inquiry; , At Such timés, it will be necessary to produce a record of our work, 
: (9 explain mU was done and provide a foundation for improvement. 
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ACH: What’s the Catch? 


It requires creativity and area 
knowledge to generate a good 
set of mutually exclusive 
hypotheses. 


It takes time to create a matrix, but it brings order 


to your thoughts and to discussions saving time 
later. 


The matrix merely makes your thinking and analysis 
explicit. This gives you the opportunity to improve, 
but diligent examination is still needed. 


15 


What is the catch with ACH? ~ 


Well, it requires creativity and area knowlédge to generate a good set of 
mutually exclusive hypotheses. 


It also takes time to create a matrix, but (on the upside) once done, it brings 
order to your thoughts and your discussions with colleagues, saving time that 
might have been spent going in circles. 


All ACH really does is make your thinking and analysis more explicit. This gives 
©,you the opportunity to improve, but diligent examination is still needed. 


15 
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: Weaknesses 


ACH does not guarantee a better answer. It is 
possible to skew the results toward a favoured, 
theory by making the other hypotheses preposterous 
or pre-selecting evidence. 


Analysts also need to be diligent at including absent 


evidence (things that could or should be observed, 
but are not — i.e. "NEGINT"). 


If one hypothesis has 10 C's and another only has 5 
C's, the former is not automatically more likely to be 
true. All C's do not have equal weight. 


ACH does not guarantee a'better answer. It is possiblecto skew the results 
toward a favoured theory bý making. the other hypotheses preposterous or 
pre-selecting evidence. For example, if yoü' waritéd to prove that Oswald shot 
JFK, your other hypotheses could be Elvis andKrusty the Clown. 


Analysts also need to be diligent at including absent evidence (things that 
could or should be observed, but are not — i.e. “NEGINT”). If, for example, a 
company claims to be legitimate it should advertise in trade publications, 
protect its intellectual property, and hire workers. If it is merely a money 
laundering front, the tell-tale sign will likely be a missing element. 


Lastly, it ishard not too automatically conclude that the hypothesis with the 
most'eviderice is correct. However, if one hypothesis has 10 C's and another 


Og... "only has 5 C's, the former is not automatically more likely to be true. All C's 
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De not have equal Weight. When the Argentinians invaded the Falkland 
i Islands i in 4982, they had lots. of evidence to suggest that the British would 


respond With, either diplomaticly or economicly. The one factor in favour of a 
military response ‘was Margaret Thatcher. As it turned out, she out-weighed 
everything else. 
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ACH: The Eig Process 


[^ Checklist 1. Identify all the possible hypotheses 
x Be sure that they'are mutually. _ 
exclusive and cover all possibilities: 


2. List all significant pieces of information and the 
arguments for and against each hypothesis. Note the 
absence of evidence. 


Create a matrix. Hypotheses in columns; evidence in 
rows. Indicate in each cell whether the information or 
argument is consistent (“C”) with, inconsistent (“I”) 
with, or is not applicable ("NA") to each hypothesis. 
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ACH has six steps. The first:three are: 


I: 


Identify all the possible hypotheses/4t js important at this stage not to 
prematurely limit the range of possible explanations. Resist the urge to 
debate if a hypothesis is likely or unlikely. If there'are more than eight 
hypotheses, try consolidating some. For example; if,you‘thought an attack 
was going to take place, but the location was unknown, instead of listing 
50 different countries as possible locations, you could just say, "The attack 
will occur in North America, Europe, or Africa." 


List all significant pieces of evidence or information and the arguments for 
and against each hypothesis. Include opinions as well as logical 
deductions. Note the absence of evidence. How? Just ask yourself, "If this 


“hypotheses: were true, what would | expect to see.” If it something is 


Paeng note it; 


OF Great a a Mon, Put the hypotheses across the top and the evidence down 
E ~ the, lef hand side. Indicate i in each cell whether the information or 


iu afgumiéntis consistent CEC) with; inconsistent (“I”) with, or is not 


applicable: (“NA”), to! each hypothesis. 
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ACH: The Eig Process 


4. Refine the matrix. 


— Remove hypotheses with lots of “l's”. 
— Remove evidence that is "NA" across the board. 


5. Draw tentative conclusions. Which hypothesis seems, 
best supported by the evidence? Which is least 


Supported? 


6. Re-examine the evidence supporting your conclusions. 
Does it all come from one source? 


4. Refine the matrix. Consider-removing hypotheses with lots of “ls”. 
Definitely remove evidence that is “NA” across the board; it is not relevant 
and not helping. | 


5. Draw tentative conclusions. Which hypothesis seems best supported by the 
evidence? Which is least supported? 


6. Re-examine the evidence supporting your conclusions. Does it all come 
from one source? What if that information were wrong, misleading, or subject 
to a different interpretation? 
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ACH: The Ei Process 


7. Report conclusions. The typical format is, “Sir/Ma’am, 
we considered eight possibilities. We were able-to 
eliminate four. At this point, we think “X” is; most... 
likely, but "Y" and "Z" are still possible. On the basis of 
the information we currently have, we have a | 
moderate level of confidence in our assessment." 


8. Identify milestones, i.e. collection gaps. What would 
prove or disprove your conclusions? 


7. Report conclusions, Thetypical format is, “Sir/Ma'am, we considered eight 
possibilities. We were able to eliminate four. At this point, we think “X” is 
most likely, but "Y" and “Z” are stilkpossible. On'the basis of the information 
we currently have, we have a moderate level of confidence in our 
assessment." 


8. Identify milestones, i.e. collection gaps. What would prove or disprove your 
conclusions? In your report, you might say, "This analysis assumes that the 
missiles have not been modified. This needs to be confirmed. If their range 
has been extended, this would have a significant impact on our conclusion." 


19 
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ACH Tool 
ACH was developed in the 1970s by Richards Heuer, a` 
specialist in analytic methodology. at CIA. 


Chapter 8 of Heuer’s book, Psychology of intaligende o 
Analysis (pdf), is devoted to ACH. 


The ACH Tool developed by Heuer 
and Xerox PARC is available for your 
desktop via the CSIS Application 
Catalog. 


How to use the ACH Tool (6 minutes) 


ACH was developed in the 1970s by Richards Héüer, a specialist i in analytic 
methodology at CIA. 


Chapter 8 of Heuer's book, Psychology of Intelligerice Analysis (pdf), is devoted to 
ACH. 
The ACH Tool developed by Heuer and Xerox PARC is available for your desktop via 


the CSIS Application Catalog. 


Click on the link to see a short video on ‘How to use the ACH Tool’. 
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ACH Tool: Example 


Audio Paying 


Thomas D'Arcy McGee, one “F 


of the Fathers of 
Confederation, died in . 
1868. ç N 


He was hit by a single shot á A 


to the head as he stood 
outside his rooming house 
in Ottawa. 


James Patrick Whelan was 
hanged for the murder. 


Let's use the ACH Tool tó examine the details of @’famous incident in Canadian 
history. ( 


Thomas D'Arcy McGee, one of the Fathers of Confederation, died in 1868. 


He was hit by a single shot to the head as he stood outside his rooming house in 
Ottawa. 


James Patrick Whelan was later hanged for the murder. 
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ACH Tool: Example 


File Edit Matix” Options Learning Aids Help : ^ 
: Type 
: Sot —— u ss 
Enter Enter i of 
- či A. 4 v T s 
Hypothesis Evidence i oe = = Calculation - consistency ils 
Type I Hi 
Killed by iui Personaf-. / 
| Fenians : Enemy 


Enter Ewdence 


i —— 
1 f Multiple Fenian death 
| | threats 


At the time, there were‘many theoriés-about McGee's death. 
Let’s consider three hypotheses: 


1) McGee was killed by Fenians, Irish terrorists whowantto, hold Canada hostage in 
exchange for independence for Ireland, 

2) McGee committed suicide, and 

3) McGee was killed by a personal enemy (perhaps someone he owed money). 


Using the ACH Tool, we enter each of these hypotheses in a separate column, H1, H2, 
and H3. 


Then we click on the Enter Evidence button and type, "McGee had spoken out against 
the Fenians and eagen multiple death threats from them.“ 


i 7 ; With; our first diece of Vide in row E1, we then work across the screen, scoring 
‘UY the evidence, against ‘each, hypothesis. The ACH Tool provides a helpful drop-down 
menu with’ Six possible choices...“ 


22 
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ACH Tool: Example 


CC (Highly Consistent): the item strongly supports 
the hypothesis being true. 


C (Consistent): the item supports the hypothesis 
being true. 


NA (Non-Applicable): the item is not relevant to the 
hypothesis presented. 


I (Inconsistent): the item does not support the 
hypothesis being true. 


II (Highly Inconsistent): the item strongly refutes the 
hypothesis being true. 


There are six options for rating evidence. The choices are: 


CC (Highly Consistent): the item strongly supports the hypothesis being true. 


C (Consistent): the item supports the hypothesis being true, 
NA (Non-Applicable): the item is not relevant to the hypothesis presented 
I (Inconsistent): the item does not support the hypothesis being true. 


II (Highly Inconsistent): the item strongly refutes the hypothesis being true. 
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Interpreting Evidence 


Interpret the term "evidence" 
Expert Advice broadly (include facts and-opinions). 


If the evidence makes you take one 
small baby-step in favour of a 
particular hypothesis give it a ^C*. 


If the evidence makes the hypothesis 
less likely, give it an ^l^. 


Evidence does not have to be 
irrefutable or conclusive to receive a 
ace 


If the evidence makes you take one small baby-stépin favour of a particular 
hypothesis give it a “C“. If the evidence makes the hypothesis less likely, give it an l^. 
Evidence does not have to be irrefutable or conclusive to receive a "C". 


24 
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ACH Tool: Example 


File Edit Matnx. mem Learning Aide Hel 


Son bo ed 


Emar Entar Evidence Order Ad ded 


Hypothesis Ewdenca 


Classification: - 


Project Tite T 


$ Death of Thomas. 
D: Arey | McGee 


Available Maeva snc are it ico EP 
Maint: š | McGee was shot while 
| opening the door of his 
1 H residence 


3 — AREE 
SH 1 
i ;Patuck Whelan’ $gunwas ` 
{| same calibre as the murder | c 
1 d weapon E I 
i i McGee was an R BISORNT i | 
m Feni ian opponent t 

+ 


i| Mati F Fenian death 


After all our evidence is‘entered; we can see that there are,two pieces of evidence 
that are inconsistent with the idea that McGee committed suicide. 


° The bullet entered the back of his neck -- awery'awkWard way to shoot yourself. 


° McGee was killed at the entrance to his residencé--a strange place to kill 
yourself. 


Since “I‘s“ are worth -1, this gives H2, the hypothesis that McGee committed suicide 
an Inconsistency Score of -2. 


Note that H1 and H3 both have scores of zero because both “C‘s“ and “NA‘s“ are 
worth zero. Sóme feel that "C's" should be worth +1, but the ACH Tool doesn't count 
them, Bretagne instéad to enamigi inconsistency. 


"Or; if you were presenting bs chatt; you would say, “Boss, I’ve considered three 
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hypotheses and fiv ve/pieces of évidence; So far, the evidence is least consistent with 
the hypothesis. that MeGee committed suicide. | have not yet ruled out that he was 
killed by Fenians or a personal enemy.” 
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“Sly 
To what extent do you think ACH accomplishs the foll 


Serve as a briefing aid? 


Provide an "audit trail“ of what was considered and how it 
was interpreted? 


Provide a useful overview (snapshot) of the evidence? 


Permit a more objective assessment of the evidence? 


Focus discussion on "diagnostic" evidence? 
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Learn more 


Visit the ACH page on 


Learn more about ACH by visiting the ACH page On), 
examples of actual analytic papers where the author used: ACH) 
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It includes links to 
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Good-bye! 


If you found this presentation interesting and 
useful, you may wish to learn more about ' 


structured analytic techniques. 


In particular, you may wish to take a look at 


the Indicators Validator, a close cousin to ACH. 


Good-bye! 


If you found this presefitatibf interesting and useful, you may wish to learn more 


about structured analytic techiniguies- 


In particular, you may wish to take a Wes atSthe Indicators Validator, a close cousin to 
ACH. 


Good-bye! 
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Version/Credits 


This PowerPoint is Version 1.0; “Ary, 
created by. on January 14, 2014. ~ Da 
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Diagnostic Techniques 
Organize / Rank / Sort / Assess / Visualize / Filter / Compare / Conceptualize / Decompose 


SIO 
` 


Sr dita A ituseful? 


Too much data? Too many theories? ACH 
organizes‘everything into a neat grid, It also 
helps mitigate the,“confirmation/bias”. 2 


- Technique Description ^ 
Analysis of 


Competing 
Hypotheses 


Bayesian 
OTT sis 


Chronologies 
and Timeline: Timelines 


Arrays evidence against rival or CORDATE? 
hypotheses. Helps disconfirm hypotheses 
based on inconsistent evidence. 


Infers the probability of an event re- 
occurring based on previous occurrences. 


Displays a sequence of events (in a list or 
graphically in a flowchart). 


Worried about “false postiyps a ane “false 
negatives”? 


Looking for Comel or patterns? A timeline 
may make relationships, gaps and y gpancies 
more apparent. JA; 


rage ainisact Examines how variables/drivers/factors How will changing one variable affect the other 
V ross-impaet ; : 
inter-relate (affect one another). variables? 


Force Field Investigates the balance of forces acting on | Need to clarify the effect of key drivers or 
Analysis issue or influencing a change. forces? 


| Gantt Chart | | Gantt Chart | | Shows the phases ofa project. =| the | Shows the phases ofa project. =| of a project. | How long will it take? How many steps? | How long will it take? How many steps? will it take? How many steps? 


Depicts hot-spots of activity using a colour Need to identify/highlight areas of concern? 
scale. 


Uses a set of observable phenomena to Need to anticipate something? I&W helps 
track events, spot emerging trends; and identify persons, activities, developments, or 
warn,of change. trends of interest. 


Indicators of 


Change (I&W) 


Indicators 
Validator 


“Limited in time or money in which indicators to 


Determines the strength Selle ney, of. an 
u /4..| watch? Need to defend the use of an indicator? 


indicator. 


"Want.to'expose and test your assumptions? 
KAC identifies collection gaps, generates 
alternative hypotheses, and exposes 

. weaknesses in arguments 


: f Breaks a predictive argument into drivers, Wantto deconstruct an argument using non- 


Link Charts Visually depicts relationships. Looking tl a way to uncover, interpret, and 
(Analyst’s display complex information in easily-understood 
Notebook) form? 


Multiple Attribute | Uses a set of defined and weighted criteria Wondering which might be the best option or 
Analysis" to compare things. highest priority? 


Path to Sorts groups or individuals into an Wonder who is most likely to commit a violent 
Viblenco* ascending series of six observable steps. act? The Path to Violence provides a good 
— visualization that is useful for briefings. 


Employs a process/to nuy. limit, check, 
assess, and revise the assumptions ~ / 4 
underpinning fundamental judgements, ,,_ 


Key Assumptions 
Check* 


Quality of “Sy, T ‘Examines the reliability of information and Unsure whether the information is accurate? 


Information 4 ^ A y the, ID of the source. 


E Forces a- móré! Gor Shpnsive 
+. S Y consideratión/of.a pM and identifies 
< ¿areas of risk. | 

SY Maps; and méasures relatioriships or links 
among individuals. groups, or organisations. 


Orders choices acearding ( to a esa 
Rankings comparison. J 
*Developed or substantially modified in Cat 


Planning a research project? Need to improve a 
collection plan? 


Need to determine who is most central in a 
network? What to know the role someone plays? 


Analysis 


Need to decide which alternative is best / most 
likely? 
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Challenge Techniques l 
Test / Refute / Invert / Expose / Examine / Articulate / Probe / Analyze / Find Alternatives 


— Technique : Description ` Rr “Ae, S - When is i ituseful? 


Višuslizəs the structure of an es Want Ap see the swap c ofan aigument? 


TN exposing co-premises masquerading as 
support elements. 


Devil's Tests an argument by attempting to prove it | Want to make your argument Mad 
Advocacy is flawed or fundamentally incorrect. 5j 


Challenges the assumptions and evidence Need to discuss a i draf paper Without jen it 
Dialectic used to frame and support an existing personal? 
Inquiry analytic position to arrive at plausible 

alternative explanations through discussion. 


High Impact — Alerts decision-makers to an unlikely event Does a neglected issue need more attention? 
Low | that would have major impact. 


Probability 


Ouitside-In Identifies the range of external forces, Wondering if there are any external factors that 
Thinking factors, events and trends that would have should be considered? 
d an impact on shaping an issue. 


Solicits constrüctive criticism. Is your paper ready for publication? 


Assumes that, In the füture, añ assessment | Worried about how your assessment might be 
has proven to be "Spectaculárly wrong” nen wrong? A shift in perspective may help. 
works backward to find the flaw, ''^ 


Pre-Mortem 
Assessment 


A team; emulating your, adversary, tries to^ ( e adversary’s tactics, techniques, and 
Red Team beat you. The-simulated conflict exposes“ 17; procedures defeat you? Want to anticipate their 
weaknesses and vulnerabilities: Un | next move? 


Compares the analysis of two independent, -| Don't want a homogenized or consensus 
teams. ;.. | Opinion? This approach highlights the reasons 
$ for disagreement. 


Venn Displays how concepts overlap, when T (Does a-"perfect storm" exist? Is there a logical 
Analysis* conditions are met or characteristics shared. | inconsistency? 


What if? Assumes that an event has occurred then Want a good technique to facilitate a group 


Analvsis works backward to explain how it came discussion? 
Anave about. 


*Developed or substantially modified in Canada. 


Team A — 
Team B 
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Imagination Techniques - new insights, alternative outcomes, and different perspectives 
Speculate / Project / Interpolate / Create / Anticipate / Devise / > Pivelop. / Generate 


Description ^ 


: Technique 


Want to stretch your amator and consider a 
wider range-of possible futures? / 


Arrays one driver of future events ágainst 
another to create a matrix with four 

quadrants, each of which is an alternative 
future. 


Alternative 
Futures 


Analysis 


Searching for- proactive’ a reactive control 
measures? 


Correlates the hazards (precursors) that 
could lead to an undesirable event with the 
consequences that could follow. 


Bow Tie* 


Asked about the future? This disciplined, scénario 
generation technique helps analysts imagine 
plausible futures and their effects. 


Identifies the drivers and assumptions that 
create a baseline scenario then changes 
the assumptions to generate two alternate 
scenarios: a plausible variation and a 
wildcard. 


Cone of 
Plausibility 


Just undertaking a study or starting a project? 
This technique creates a high-level overview of 
the key concepts and their interconnections. 


Displays the relationships between 
concepts. 


Concept Map 


What will B look like if A happens? If the problem 
can be put in these terms, counterfactual 
reasoning can help you to better understand 
precursors and potential outcomes. 


Forces a more complete consideration of an 
outcome that iS Conditional on a previous 
event. 


Counterfactual 
Reasoning 


Establishes chains of decisioris;and/of 
events which illustrate-a comprefiensive, ; 
range of possible: “future actions. [ 


“Trying to imagine possible permutations of a 
“}\process? Want to create a model for how events 
44>) could-unfold. 


Decision Trees 


Wantto. capture suggestions and critical 
comments anonymously? Through multiple 
iterations this technique finds a consensus. 


Force Field Visualizes the forces for and against à a Need to ete represent the forces at play? 
Analysis _ Sis change proposition, problem, or goal. i 

Hypothesis Provides a structured mechanism for Want m broaden your thinking? 

Generator generating a wide array of hypotheses. 


Hypothesis Mitigates mirror-imaging bias and helps What are the alternatives from the adversary's 


Produces reliable consensus;judgements 
from a large and diverse set of experts. 


Delphi Method 


Review understand an adversary's potential point of view? 
Technique mitigation strategies. 
Issue peus clarify the needs of a client. Is the question or problem unclear? 
Redefinition 
Linchpite Structures a predictive problem by defining | Want to focus your analysis on the key drivers 
uy, / ? 
yeas sis ~V/o >the factors which drive and influence a shaping an issue? 


ON _Sitéation. 


CE ;Amproves critical and creative thinking by 
depicting relationships visually. 


Having difficulty visualizing a problem or topic? 


Want a structured approach to elicit ideas from 
participants, facilitate group discussion, and to 
produce a rank ordered or prioritised list of 

ideas? 


OU » ON , S EN Generates hew analytic ideas, hypotheses, 
Nomiftal/GroupS En OF opes. 
Technique ^O, a Ae. ^ D 


Gene rates four main hypotheses for a topic 
or event. 


Quadrant 


Hypothesis 
Generation 


Technique 


How do the two key drivers shaping an issue or 
situation interact? 
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Invents plausible scenarios using an 
intemal think tank created to help develop 
broader peripheral vision and avoid 
surprise. 


Red Cell 
Red Hat* Provides a deeper and better understanding Want to understand your adversary from their 
(Anthropology) of a group, tribe, or organization. ^. / perspective? Baffled by “irrational” behaviour? 


Red October Assess both the feasibility and impact of an ^| Want to-anticipate threats but be realistic about 
maa RM adversary's actions. 


Structured Generates new ideas and concepts from a 
Brainstorming group using divergent-convergent thinking. 


feasibly and Consequences? 
*Developed or substantially modified in Canada 


Worried about being blind-sided by an issue or 
‘problem no one ever considered? 


Want to usé á guided group discussion to 
broaden your thinking?,.. © ©~ 4 
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In 2007, Structured Analytic Techniques (SATs) wére identified as one of the Eight Best Practices 
of Intelligence Analysis by Canada’s Intelligence Assesi orit ap ating Committee. 
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D103 


Structured Analytic Technique 


Alternative Futures 
Purpose 


To systematically identify a range of alternative 
futures, monitor for signposts, and thereby avoid 
surprise. 


Description 


A group of experts collectively identify a set of 
drivers (forces, factors, or events) that are likely 
to shape an issue. 


During the “quadrant crunching” phase, drivers 

are arrayed in 2x2 matrices where each 

quadrant is a different “future world”. E.G. 
Improvised Explosive Device (IED) Attacks 


Skilled 


Rare but 
deadly 


Dalang 
5 anpa 


Supplies 
Scarce Plentifüf 


Caii and Hit and 
Quiet Miss 
Unskilled 


Analysts then ask: What signposts or indicators 
would exist on the path to each "future world"? 


Usage 
Mediu and long-term strategic scenarios. 


Requirements . > Of 


^ Ta 3 


^re drive are. usta generated through a 
structured. EENETI session. This provides 
a rich, full: set. | 


The quadrant crust ae ean be more 
quickly accomplished by dividing the.group into 
break-out teams, each with a paired set óf 
drivers to consider. 


Strengths 


Most other efforts to'anticipate future events 
usually end up with minor variations of current 
trends: A» 
"Across the Gee /nment, m 
were failürés;of imagination, policy, 
capabilities, and management.’ 3 


- 911 Commission Report. 


The Alternative Futures technique forces 
analysts to look at not only the most likely 
scenarios but also those that might seem 
unlikely, even counter-intuitive. Some of these 
may have high impact. 


Weaknesses 


A highly efficient and effective technique for 
generating an extremely broad set of 
alternatives when faced with very little data and 
high, degrées of uncertainty, but ... 


« / dt takes time and effort to do well and 
wpa Sey pe group of participants is critical. 


Also, its difficult.to manage more than eight 
drivers. Fourdrivers can be combined six 
different ways, six drivers can be combined 15 
ways, and eight drivers 28 ways. Fortunately, 
drivers can usually be consolidated to simplify 
the exercise. 


Theory put into Practice at Meech Lake 


On October 31, 2007 over 30 intelligence _ 
analysts from nine different departments and ` 
agencies attended an “Alternative Analysis ` 
Exercise” at Meech Lake near Ottawa. The 
event was a deliberate mix of subject-matter _ 
experts and analysts on the periphery ofa 

_ key security issue for Canada. One veteran - 2t 
analyst commented that, despite intimate 
familiarity with the topic, the exercise forced ` 
him to consider a fuller range of passible e 
outcomes. S c 
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Structured Analytic Technique 


Analysis of Competing 
Hypotheses (ACH) 


Purpose 


To systematically evaluate evidence against 
rival or “competing” hypotheses and thereby 
avoid the natural human tendency to “anchor” to 
a favoured hypotheses. 


Description 


Human beings have an innate tendency (a 
“cognitive bias”) to jump to conclusions. 
Investigators and analysts are no exception. 
They are quick to form a hypothesis. This is not 
a problem except when evidence is ignored or 
excluded because it doesn't fitthe hypothesis. 


This "satisficing" approach risks the “confirm . 
ation bias" — selectively hearíng or seeing Only. 
what supports a preformed judgment or opinion: 


ACH arrays the evidence against multiple 
hypotheses to find the best fit and discoónfirm 
competing hypotheses. 


Experienced investigators and analysts do this 
to some extent already, but ACH does a great 
job of organizing the evidence and forcing a 
more systematic evaluation. 


Usage 
ACH is best suited for tactical analysis. It 


struggles with strategic analysis because the 
eviderice ii is. more: eee 


»,| Requirements., 


A 
f 
UT. 
ud P «P 
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Oné Simply places ny Vinee in-columns and 


evidence i in‘rows, ‘Creating a gridcor matrix, Then 
one scores‘éach piece of evidence as either 
consistent, inconsistent! or, neutral with each 
posited hypothesis. - 


This can be done in MSWord in a table, using 
MSExcel or with the ACH Tool, free software 
developed by Richards Heuer and Xerox Parc. 


® tttd © 
Strengths 


In theory, ACH encourages a more objective 
assessment of the evidence. 


An ACH grid próvides a useful snapshot that 


- summarizes all the thinking-(not just the 
evidence in support of the: IER, 
hypothesis); 

= Serves well as a briefing aid; 


= helps identify where opinions differ in the 
evaluation of evidence; 


= provides an audit trail of what was considered 
and how it was interpreted. 


Additionally, the ACH Too! can sort evidence by 
"diagnosticity." This helps analysts focus on 


‘Sal evidence that discriminates between the varying 


hypotheses. It may reveal, for example, that the 
leading hypothesis is mainly supported by 
evidence from a single human source of 
unconfirmed reliable. 


Weaknesses 


ACH does not guarantee a better answer. It is 

possible to skew.the results toward a favoured 
theory by Sétting up “straw man" hypotheses or 
pre-selecting evidence. 


Success depends on creatively brainstorming 
multiple hypotheses that are mutually exclusive. 
The former requires time, while the latter may 
be unrealistic in that several things may be 
happening at once. 


Analysts also need to be diligent at including 
absent evidence (things that could or should be 
observed, but are not — i.e. “NEGINT”). 


Further Reading: 


* ACH was developed in the 1970s by Richards Heuer, 
a former CIA Directorate of Intelligence methodology 
specialist. Chapter 8 of Heuer's book, The Psychology 
of Intelligence Analysis, is devoted to ACH. 


To download the free ACH Tool, go to: 
http://www?2.parc.convistl/projects/ach/ach.html. 


UNCLASSIFIED 


3103(18/09) 


Structured Analytic Technique 


Argument Mapping 

Purpose 

To better understand the structure of arguments. 
Description 


Argument mapping is, roughly, making a picture 
of reasoning. More precisely, it is the graphical 
display of the structure of reasoning and 
argumentation (see Austhink video). 


Typically, argument maps are box-and-arrow 
diagrams, a bit like flowcharts. 


Argument mapping belongs to a family of 
"thought mapping" techniques which includes 
concept mapping and mind mapping. 


Usage 


Argument maps can be ieee to Help write an 
essay (see video) or visualize a complex\s 
argument made in court. 


Requirements 


A typical argument map starts with a statement, 
E.G. “Deploying UAVs along the border will make 
us more secure.” This is contained in a box at the 
top of the page. Underneath, boxes are added 
containing evidence or arguments that support 
and oppose the statement. 


Yu SG aaa Tas .` 
Deploying UAVs along the 
border will make us more 


UNS a E 7 
J/-3oompared to ather l ` 
options; ! 


More surveillance 


JM All the have. 
means more security 


not TO God im «x 


Strengths 


Software-aided argumentmapping enables 


|'analysts to rapidly build; modify, Vigw s and 


share their Work. 


Rationale 


` a| Common errors such as confusing helping 
&d poeni ises, for separate reasons are easy to 


panes evidence Suggests that it can help 


| "improve, reasoning. Tests on students at the 


University of Melborne showed improvement of 
0:8'of a standard deviation as measured by the 
California Critical-Thinking Skills Test. Another 

study found three times the gross gain typical 
of other endeavors. 


Weaknesses 


The best results require intensive practice on 
computer-supported argument mapping in a 
highly-structured setting. 


The approach is surprisingly time-consuming. 
It is difficult to know when to stop. 


It may degenerate into a game of Boxes-and- 
Arrows, if insufficient effort is put into thinking 
about why, or how, or how likely one thing 
connects to another. 


Further Reading 


s Argument Mapping Tutorials — from Austhink. 
= Argument Mapping Intro (video) Kookaburra Case 
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Structured Analytic Technique 


The formula tells;us that if we see the indicator, 
there is an 81% chance of war. The presence of 
false positives and false negatives has 
downgraded the reliability of the indicator. 


Bayesian Analysis 


Purpose 


To assesses the likelinood of an event or 
reliability of an indicator. 


Usage 


Use Bayes whenever thé question is: “How 
likely is X, given Y". This often’eccurs/in-cases 
with indirect (circumstantial)! evidence; 


Description 


In Bayesian analysis, the probability of an event 
occurring in the future is inferred from previous 
occurrences of the event. 


Requirements 


The analyst must accurately estimate the 
probability of events. This requires good data 
and reasonable assumptions. 


Say our neighbours have moved troops to our 
border 1000 times, and on 100 of these 
occasions actually invaded. On 90 of these 
occasions, they moved supplies to the front 
lines before they invaded. Most;people think 
that this indicator is 9096 accurate because it 
was present 90 out of 100 times before war > ` 
occurred. They fail to"'includethe false positives, 
E.G. the 90 occasions when the indicator was 
present but NO invasion followed: They also fail 
to include the false negatives, E.G;the 10. '' 
occasions when the indicator was absent,but-an 
invasion occurred (see table below). 


Strengths 


=b This technique helps validate the reliability of 
indicators, providing numeric values. 


‘ithelps'counter the bias in favour of new or vivid 
| evidence... 


Weaknesses 


The calculation of a priori probabilities is critical. 
Miscalculation, biases the final result. 


Indicator 

x __|NOT present 
— true positive 
No War | 90 -false positive 


To find the actual reliability of the indicator, we 
use the following formula: 


The math can be intimidating for some. 


Further Reading 


* Bayes’ Theorem for Intelligence Analysis, (1972), by 


Vnd aiiud ie ag ne Jack Zlotnick in H. Bradford Westerfield, ed. Inside 
P(BIA) is the-conditianal It is “prior” in the sense that it CIA’s Private World: Declassified Articles from the 
probability of B given A. does not take inte account any Agency's Internal Journal (New Haven: Yale University 
‘OYA On information about B. Press, 1995). 


' C Vf , "^ 

Bus ER r4 EOS V^. e Fisk 1972. The Sino-Soviet border dispute: a 

ALB) s. PURO. CA) ^ comparison of conventional and Bayesian methods for 
UON «5 £y, PCB) SA, Intelligence warning. Studies in Intelligence 16:2 53-62. 
A. “OF Ep “Ss P(BYisthe Briof or marginal 


\ * Loy TENS, q PRONUM UE e Schweitzer 1976. Bayesian analysis for intelligence: 
P(A|[B) is the conditigpal probability ~ ¿ i .20: M 

of A given B. Tt is also cabbiighe VS) Ns some focus on the Middle East. 20:2 31-44 

posterior probability because its On EA , . i š 
derived from or depends upon the © , Bp LA e Captain David Lawrence Graves, ISAF, Bayesian 
specified value of B. - ! Analysis Methods for Threat Prediction, MSSI Thesis 


(Washington: Defense Intelligence College, July 1993) 


"has. 
P | Z Ri 


“A” is the 180 (90 + 90) times the indicator is observéd 
“B” is the 100 times war occurred 


15 
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Bayes in Court 
Bayes' theorem is used in court cases to Ear statistical evidence, 


However, in 2010, a case in the UK fell apart on appeal wheri the judge debited the; — did 
not provide a sufficiently reliable estimate of probability: The problem was insufficient datàc, Sus 


The case involved a Nike shoeprint that was found at a murder scéne British: police. were able to 
match it to a suspect, but the issue brought up on appeal was the reliability of, the match. 


At trial, evidence was presented that between 1996 and 2006, Nike distributed 786 600 pate of 
trainers. On this basis, a calculation could be made on the likelihood of the suspect having the ` 
same shoe as the killer. 


However, several questions immediately arise. 
e Why use 1996 as the starting year? Why not 1995 or 1997? Why not start in June of 1998? 


° Does Nike really know/precisely how many shoes they distributed? How many were actually 
sold? Does this include online.sales? 


e Are there other. shoes not manufactured by, Nike which have similar soles? What about 
counterfeit Nike shoes? 


e What is the population of the immediate area in which the suspect lived or murder took 
place? Should the “immediate area” bé:one'block or a square mile? 


The decision on how to “bound? the initial dataset Has huge Ti Ahications for the reliability of the 
result. 


No one knew exactly how many Nike trainers of the type worn by the killer were in the country at 
the time of the murder so the judge decided that Bayes' theorem shouldn't be used. 


But just because there isn’t a precise count, doesn’t mean that the result is worthless. Numbers 
that are unknown can be estimated. As long as these estimates are reasonable, the result will be 
reasonable. 


In the áftermath.of the trial, it was reported in the media that the judge had banned the use of 
Bayesian analysis: imthe court. Not so. He banned unwarranted precision. He felt that Bayes 
E i on gave a qued sense of precision if the underlying information was not ‘firm’. 


S i (os the: cprosecutis itm his case d Dyerlaved his hand in presenting Bayesian analysis as 
definitive’ Had he, been moré cautious; using it as just one several means of drawing a connection 
betweer the; tege. and the suspect, itwould not have been the focus of the case and become 
such an issue: MG 


In the end, this case is Consors S tale for the appropriate use of probability and statistics. It 
reinforces the need to be honest about limitations. It also reminds us that complicated explanations 
can be a house of cards, impressive but flimsy: 


16 
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Structured Analytic Technique 


Bow Tie Requirements 


Purpose In-depth area knowledge. 


Imaginative thinking. 


To examine precursors and consequences of 
undesirable events to develop means to 
reduce risk and enhance recovery. 


Broad perspective (possibly aided‘by outsiders). 


Description Strengths 


Once hazards and consequences are defined, 
the impact of both proactive and reactive control 
measures can be considered. 


This technique correlates the hazards 
(precursors) that could lead to an undesirable 
event with the consequences that could follow. 


Proactive controls reduce or eliminate hazards. 


Proactive Controls (Treatment) 


ndesirakle Event Reactive Controls (Treat ment) 


Reactive controls reduce the potential impact of 
the event and assist in the recovery from the 
The Risk | š consequences. 


Event 


Weéaknesses 


- | Órílyywofks;on risks which have been identified. 
Consequently, the analysis will only ever be as 
good as the risk identification process. 


The technique also allows the development 
and consideration of both proactive and 
reactive control measures, and can show the 
effect of a policy, or physical change on the 
issue. 


? à Bow-Tie Analysis 
na onze ed Computer Access" 


Undesirable Event. An unauthorized 
person gains access to our internal network. - 


Usage 


Hazards: An intrusion could occur because ` 
we have not changed default system ` 
passwords, new patches have not been 
installed, an ex-employee has retained 
access, etc. 


Begin by identifying an undesirable event. This 
could be anything from the failure of policy to 
the rise of a dictatorial leader. 


Then identity, hazards that could lead to this 
event. Hazards: tan bé factors or drivers — i.e. 
circumstances; facts Of; influences that, either 
“separately. or in combination could cause the 
event to' take place. Depending on ithe 
numbef of hazards identified, a process of 
ranking them s Spy likelitjood or im ages Zmay 
be necessary.  '444^. ! 


* Proactive measures: check settings, 
download updates, coordinate with 
personnel, etc. 


Consequences: loss of data, public ` 
disclosure, extortion, damage to ` 
etc. 


Finally, identify the consequendes.of the. . 
undesirable event - physical, psychological. 
internal or external, etc. It may be useful to 
separate them into categories or rank them in 


- Reactive measures: crisis 
communications policy, insurance, 
business continuity plan, etc. - 


17 
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Structured Analytic Technique 


A N 2 


Brainstorming 
Purpose 


To generate new ideas 
and concepts from a 
group using divergent-convergent thinking. 


Description 


Structured Brainstorming is a two-stage group 
activity that takes 60-90 minutes. 


In the divergent stage, each participant writes 
his or her ideas on a sticky note. Throughout 
the session, the facilitator collects these. As he 
randomly places them a board in full view of 
everyone, he reads them aloud’to.the group. 


In the convergent stage, these ideas are i 
rearranged and clustered-into key themes by, , 
the facilitator or designated members of the 


group. 
Usage 


Brainstorming is most beneficial at the start of a 
project. 


Requirements 


Ideally, participants should come from different 
backgrounds and hold different points of view. 
This stimulates discussion. 


It can. Speed up the process if you prepare a 
briefing Gn,the topic and distribute it beforehand 
or present, it at kee Stat of the session. 


A (Ask thes participants to focus off & topic or 
> “question, that is,neither too-broád nor tóo 
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narrow, Write it RiSininentiy at the front of the 
room. ( ! 


Provide each partici GAGs wifi a (pad. of; Post-it ™ 
notes and a Sharpie™ marker: 


Brief them on the process that you will follow. 
(aims, objectives, stages, timings, breaks, etc.). 


During the divergent stage, encourage 
participants to listen carefully and think 
creatively” They must Suspendjudgement, and 
resist the temptation to self-Censor-or be critical 
of others.<No idea Is+oo weird or wild! What 
looks crazy at first may,become valuable later. 


Expect a lull after the initial round.of 
suggestions. Allow up to 5-10 minutes ‘of “dead 
air" for people to reflect on what they-Fiave 
heard. Eventually, a second round of 
suggestions will begin. 


During the convergent stage, it is typical for 40- 
60 suggestions to cluster into 5-8 themes. This 
often involves some debate. Be flexible. 


Discuss outliers before eliminating them. 


)A trained facilitator can help ensure a fruitful 
-seSsion, especially with groups larger than 6 or 
»f people. Setting the right tone is key. 
| Strengths > 


~..| The-group dynamic stimulates the imagination. 


One person's thoughts may trigger an idea for 
anotherperson, 


The divergent phase encourages spontaneity 
and exploration, while the convergent phase 
provides focus. 


This method generates many ideas very quickly. 
It is an effective way to broaden your thinking. 


Briainstorming is often used in conjunction with 
other SATs such as ACH, Red Cell, and SWOT. 


Weaknesses 


Office politics, poor group dynamics, 
participants with an agenda, or uninspired 
leadership could sabotage a fulsome session. 


Some people need quiet time to reflect; they 
don't do well in group situations and can’t think 
on their feet. One solution is to provide breaks 
or schedule the sessions a week apart. 


UNCLASSIFIED 


Structured Analytic. Technique 


Chronologies and Timelines 
Purpose 

To organize or display data by date and time. 
Description 


Chronologies list events in the order they 
occurred, usually in narrative or bulleted form. 


Timelines arrange information graphically along 
a chronological spectrum. 


| Account 
LOpened | 


Usage 


At the beginning of an investigation, à chrohól». <] 


ogy of events or timeline can be useful to 
sorting the data. 


During the analytic process, newly acquired 
qata can be added to chronologies and time- 
lines as events happen or when additional 
information comes to light. 


This is also when they can reveal gaps, 


discrepancies, and patterns such as “command 
and control” structure. 


Who i i$ s giving ihe. orders? 


In the final product, chronolagiag 4 time-lines 
are particularly useful for displaying the data for 
consumers. 
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Requirements 


Ideally; the, date and time trust be: known for 
each event, The data must then be Apyied. 


A chrono is usually a simple list of arenis: 
either in a table or äs: bullets: 


Timelines usually proceed front left ó Pent 
Software, such as Analysts Notebook , can be 
used to generate timelines. So can RFFlow and 
SmartDraw. 

Strengths 


Chronologies and timelines: 


help sort and manage data; 


may reveal relationships and patterns that 
/,,, dfen't. apparent in regular narrative text; 


“emake flows more explicit; 


discover gapSand generate requirements for 
‘additional research or investigation, and 


314 - ald ‘readers to understand. 


In certailt situations, a timeline can be a 
predictive tool. For example, knowing what 
stage an adversary is at in a process may 
provide an indication of when the process will 
be completed. 


Weaknesses 


Noticing a gap or discrepancy requires logic, 
common sense, or subject matter expertise, i.e. 
an intimate knowledge of the steps in a process. 


It is difficult to notice the absence of evidence 
(the dog that didn't bark in the night). 


Detecting patterns in data sets involving 
hundreds or thousands of events usually 
requires specialized software. 


UNCLASSIFIED 


Structured Analytic Technique 


Concept Maps 
Purpose 


To visually represent knowledge with emphasis 
on the relationships between concepts. 


Description 


CMaps aim at developing new insights. The 


technique is primarily “Imaginative” although it is 


arguably also "Diagnostic". 


CMapping is a way of 
knowledge diagramming. 


USE 
A CMap is a network of linked by 


nodes. Each node represents $ 


a concept. The relationship 3 
between two concepts is Words 
described on the directional 

link between them. This,node-, SE -Y 
link-node unit is called a IS ipa. 
proposition and is read in the WE 


direction specified by the | Proposiions dh 


arrow. 
Both concepts and relationships require clear 
articulation if the propositions that comprise the 
map are also to be clear. 


An entire CMap is intended to provide a 
comprehensive answer to its Focus Question. 


Although similar to mind maps in some ways, 
Cmaps are networks, not strict heirarchies and 
includé'tmore than just one idea. In addition, 
through labeling. ofthedinks, CMaps allow many 


E nds of rea to be a ressed, 


£1 ie Usage. 
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CMaps are, BEES Sallie "c at the start of 
a research project , to Organise ideas and at the 
end of a project to validate: the, thinking process 
and identify any remaining gaps: 


Requirements 


CMaps may be constructed individually or 
collaboratively. “The steps involved: are: 


* Formulate a Focus Question; 
= Generate a list of relevant concepts; 


= Begin connecting concepts. with explicit 
linking words and add concepts to: the» 
structure; 


= Periodically consider relevance of included 
concepts, relationships, and concept 
organization to the posed Focus Question; 


s Consider addition of arrows and other 
concepts; 


* Reposition and refine map structure and 
ensure that the map answers the Focus 
Question. 


To see the above list as a CMap, visit IHMC. 


Strengths 


Maps stimulate creative insights; encourage 


' hon-lineac thinking; clarify meanings; 


externalize-and.decompose knowledge; and 
promote relévancy by requiring that map 
elements collectively provide an answer to the 
Focus Question. 


Most importantly for analysts, CMaps reveal 
gaps in knowledge and differences in 
understanding, thereby guiding research, 
promoting discussion and facilitating 
collaboration. Ultimately, they produce a better 
shared understanding. 


Weaknesses 
CMapping may not suit non-visual learners. 


Further Information 
= CMaprTools — 


* The theory underlying concept maps and how to 


construct and use them. Novak, J. & Canas A.J. 


free software by IHMC. 
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UNCLASSIFIED 


Structured Analytic, 


Technique 


Cone of Plausibility Requirements 


A good ungerstanding of the subject matter is 
needed to select.the relevant drivers and a good 
imagination helps in-working through the impact 
of changes in assumptions, 


Purpose 


To generate a small set of scenarios. 


Description 


Results are better as a group exercise. 


The Cone of Plausibility is a scenario generation 
technique. It produces a set of plausible 
Scenarios or "futures", typically three: baseline, 
plausible variation, and wildcard. 


Strengths 


Greater simplicity and speed relative to other 
scenario generation techniques. 


;Piausible 
alternative 


The technique encourages analysts to 
challenge conventional thinking. 


> Baseline 


It is fairly intuitive and commonly used, 
especially in the military. It does not require 
b elaborate explanation. 


Wasara. 
Steps: `S 


* Choose the timeframe. How far into the “VO “Te process easily translates into a written 
future do you want to do? Weeks, months, | report, "We considered several drivers and 
or years? made anumberof, assumptions to generate the 


* Identify the relevant drivers (at least 3 but V flowing three possible scenarios.” 


no more than 5-7) that will shape events 
over the selected timeframe. 


It can ‘sensitize analysts to the kinds of 
signposts, and)indicators they might see, if a 
given scenario, was actually beginning to 
emerge. This makes the approach useful for 
identifying Indicators and Warnings (I&W), and 
ultimately providing customers with Strategic 
Early Warning. 


* Make assumptions about how each of these 
drivers will behave (one assumption per 
driver). 


= Generate a baseline scenario (usually an 
extrapolation of the present day situation). 


= Generate a plausible variation by 
changing the weakest assumption. 


Weaknesses 


It oversimplifies the future. There could be more 
than three possible scenarios. 


a Firially,. create a wildcard scenario (one 
that would, have high impact) by radically 
Rum an esum ption. 


This technique is in the imaginative category; it 
generates scenarios. It isn't intended to be 
predictive; it doesn't assign probabilities. 


If dde at ihe diy io a T" this PL 
can shape collection efforts. 


Analysts need to be careful to present the 
results as future outcomes that are plausible 

under certain conditions, rather than as what will 
happen. 


It can be used to test the impact of change. 


The “scenarios” generated are useful for: 
contingency planning. 
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UNCLASSIFIED 


Structured Analytic, Technique 


Counterfactual Reasoning Graphically, the process looks like this: 


7 > 


Purpose Intermediates 2 


To force a more complete consideration of an 
outcome that is conditional on a previous 
event. 


» S 


Antécedents 


Description Timeline’ 


LS 
Future 


YOy- 


Today 


Counterfactual reasoning poses a hypothetical 
“What if?” question. E.G. If I were to buy a car, 
what would that car be? Note that two events 
are mentioned (a decision, Event A, and a 
purchase, Event B). The idea is that B could be 
affected by how A comes about and by 
background factors affecting both A and B. 


Historians engage in counterfactual reasoning 
when they posit an alternate history, E.G. “What 
would the world look like today if Napoleon had 
won at Waterloo?” 


Requirements 


Usage — Se | A good understanding of the context of the 


situation / problem and knowledge or related or 
historical examples vastly enriches the process. 


“What will B look like if A happens?" 


If the problem can be put iA these terms/use 
counter factual reasoning to better understand 
precursors and potential outcomes. 


Stréngths. | 


This: technique prompta analysts to consider the 
Vos ways (and reasons) an event could occur. 


1. Identify the possible ways that A could come 
about (“antecedent scenarios”). E.G. | might 
buy a new car because my old car broke 
down, | want the latest model, or my needs 
have changed. 


It shifts attention from outcomes to the 
precursors and/background factors that 
influence events. 


Weaknesses 


. Rank order these scenarios in terms of 
probability, i.e. Which is most likely? 


As with any hypothetical exercise, there is no 
guarantee of accuracy. Factors can be missed 
or mis-weighted. 


3. Consider background events or factors 
(‘intermediaries’) that could affect A and B. 
E.G. IS‘Myjob’Secure? Do | have other 
major expenses? Is Oe car | want available? 


The model created by this process is a 
simplification of reality. It may give analysts a 
false sense of confidence that they truly 
understand the dynamics of a complex problem. 


a Rank: ‘order these intermediaries i in.regard to 
&< “their Potential j im par onA and B. 


5 tiis poss (fies. (‘Consequent scénarids”) 
for B. What are. the Options? E. G. | could buy Source: Counterfactual Reasoning - A Basic Guide for 


Analysts, Strategists, and Decision Makers (pdf) by Noel 
a Mustang, Corvette, or, Porsche. Hendrickson, Ph.D. with thanks to Gudmund Thompson. 


6. Rank these scenarios. Which ig most likely, 
given the ways A could occur and ` 
background factors? 
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UNCLASSIFIED 


Analysts Go Counterfactual 


On April 9, 2010 analysts from CSIS, ITAC, 
DND-CDI, PCO-IAS, IDRC as well as Prof. 
Matteo Legrenzi from the University of Ottawa’ 
participated in an exercise organized by EDC’s 
Political and Human Rights Risk Assessment 
Department (PHRAD) that used “Counterfactual 
Reasoning” to examine the impact that potential 
state failure in Yemen could have on the long- 
term political and economic stability of Saudi 
Arabia. 


The exercise was coordinated by John Bitzen 
from EDC and facilitated by Gudmund 
Thompson from DND-CDI. 

Why Counterfactual Reasoning? 


Counterfactual Reasoning exercises provide 


analysts with a process for anticipating the likely. | 


causes and consequences. of potent high in ime 
pact future scenarios. | 


In the case of EDC, the goal was to understand 
what Saudi Arabia would look like if Yemen 
were to experience a complete collapse of 
central authority or state failure. 


What was the process? 


In every counterfactual reasoning exercise, 
participants first brainstorm and then rank, 
according to criteria agreed upon by the group, 
the antecedents that could bring about a 
potential high-impact “what-if” scenario. 


For the. group, this meant brainstorming 
developménts thát-could bring about the 
collapse of; thé Yemeni state. 


= The key scenarios that were identifi ed as being 


most Jikély to fead to state Collapsejin Yemen 
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were a: ‘succession Crisis; a successful rebellion 
by southern ‘secessionists: that challengés the? 
ability of the Yemeni! central, government to 
control key oil-producing areas’ and à major 
economic crisis that sparks so¢ial-unrest,, 


Having identified a list of scenarios believed 
most likely to 1ead to.potential state collapse in 
Yemen, the next phase of the exercise involved 
generating a list of./"intermediate.events" that 
are independent of the “what if';scenario but 
coüld,still Have cónsequences for the topic 
being analysed) 


In the context of the exercise this meant 
identifying factors that aré unrelated) to the 
scenario of state collapse in Yerríén;but Would 
still influence Saudi Arabian stability. The 
discussion highlighted the price of oil and 
changes in US foreign policy in the Gulf region 
as two factors that would have the greatest 
potential impact on Saudi Arabia's political and 
economic stability. 


After mapping out the scenarios most likely to 
lead to Yemeni state collapse, and with the 
intermediate factors likely to influence Saudi 
Stability kept in mind, the group moved on to the 


"final Stage of the exercise: the consequence 
:assessmeht. 


The group’s bottom line conclusion was that 
even a complete failure of the Yemeni state 


would be'unlikely to have a significant impact on 


Saudi Arabia's economic and political stability. 
This conclusióniwas based mainly on the 
assessment that, Saudi Arabia had the means 
and historical experience to counter any security 
threats that could arise from state failure in 
Yemen. 


How useful was Counterfactual Reasoning? 


The ultimate conclusion was not radically 
different from preliminary assessments but 
going through the exercise exposed everyone to 
a wider range of views and to challenge 
questions. It also prompted useful discussion 
about the concept of state failure and helped 
flag other scenarios unrelated to Yemen, such 
as a regional war involving Iran, that are likely to 
have a greater impact on Saudi Arabia's 
economy and security. 


Reprinted with permission from the May 2010 edition of 
the Intelligence Analyst Training Newsletter. 
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UNCLASSIFIED 


Cross impact Analysis 
Purpose 
To highlight how factors inter-relate. 
Description 


It is typical for analysts to examine the factors or 
drivers of a situation in isolation. Analysts will 
systematically move from one factor to the next, 
thinking about how each one affects the 
eventual outcome. For example, poverty and 
lack of education both contribute to crime. 
However, many "variables" are not completely 
independent-their affect can strengthen or 
inhibit other variables. A lack of education, for 
example, makes poverty móredikely. Cross 
Impact Analysis forces analysts to Consider.how 
the relationship between variables could‘ affect 
the final result. : j 


Usage 


Cross Impact Analysis is useful in the early 
Stages of a project. Once a brainstorming 
session has determined all the variables, 
drivers, or players that may influence the 
outcome of a situation, the next step is Cross 
Impact Analysis to systematically examine the 
relationships between each of the variables. 
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e! 


Use this technique whenever you need to know: 

e how all the factors relate to and influence 
one another; 

e stabilizing and destabilizing forces; or 

e implications (2nd order and 3rd order 


effects), 


The basic steps tó-Cross Impact Analysis, are: 

° Identify the setofrelévant variablés: 

° Create a matrix by. listing. these variables 
down the left side andsacross;the top, 

e Work across the first row. Ask, Does, the 

presence of Variable 1 increase ór decrease 

the influence of Variable 2, 3, 4, etc.?” 


Structured Analytic Technique 


-| positive, — negative, — — strong negative 


1 The félationship between variables is assessed 


e Record the magnitude of this effect in the 
appropriate cell. If one variable is a catalyst, 
enabler, or force multiplier for another, use a 
plus sign (“+*),to code theeffect.as 
“enhancing’s UJseé-a minus sign if the effect is 
"inhibiting" and NA f there is no effect. 


For example, wealth and education correlate 
with less crime. The relationship.betwéen these 
variable is reflected in the matrix below by'a 
plus sign (highlighted in red). 


Neighbourhood Crime 


Gangs 
[Gans | NA | - f 


Legend: NA means "no affect", + positive, ++ strong 


"twice; because it may not be symmetric. 


Strengths | 


Finds thémost influential variables and exposes 
variablesithat were thought to be independent 
as inter-related//interdependent. 


May reveal where a confluence of events or 
factors create a “perfect storm”, i.e. a 
combination of variables that reinforce one 
another (a positive feedback loop), leading to 
rapid change. 


Provides a first step toward a Causal Loop 
Diagram and quantitative modelling. 


Weaknesses 


This technique can be time consuming if all 
possible interrelationships are explored. It also 
relies heavily on the opinions and knowledge of 
the analysts involved making it difficult to 
reproduce results. ` 


e Structured Analytic Techniques by Pherson and Heuer 
has a chapter on Cross Impact Analysis, 


e An Elementary Cross-Impact Model, RAND. 
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UNCLASSIFIED 


Structured Analytic Technique 


Deception Detection 

Purpose 

To avoid manipulation by an adversary. 
Description 


Adversaries use Denial and Deception (D&D) to 
conceal their activities and intentions. It 
enhances effectiveness and enables surprise. 


= Denial refers to blocking access to useful 
information. 


* Deception refers to making the target 
believe something that is false. 


D&D occurs at all levels. Tactical deception 
might involve leaking information.that an attack 
will occur at a false time and place. Strategic 
deception is aimed at high-level decision 
makers. During the Cold, War; for PXemple, the 
US deliberately leaked documents to thé“ 
Soviets about computer chips arid military; 


s Costs — Can deception be accomplished at a 
reasonable cost? 

* Feedback — Can the'adversary observe the 
effects of deception? 


2) Would deceptionibe consistent with Past 
Opposition Practices VORE L 


s Does the adversary have a history t ot 
deception? 

* Does the observed activity fit a past pattern of 
deception? If not, are there other historical 
precedents? 

* Have circumstances changed that would alter 
the form of deception? 


3) Is the Manipulability of Sources (MOSES) a 
concern? 


) # Is the source of information reliable? 
/;*^ Does the source truly have access? 
b>" How good are the sources bona fides? 


* |sthe source vulnerable to control or 
manipulation by-a foreign agent? 


technology. The Soviets invested heavily.only fo acd 


find the technology was unworkable. 
Usage 


Analysts should look for deception whenever 
they are faced with a sophisticated adversary 
(i.e. one that knows your collection capabilities 
and the indicators you monitor). 


Requirements 


When déceptiori is eo there are four 
questions. 1o. ask: 


A y Dogs the add have thé Motive, 
à v Opborturity, and Means (MOM) to deceive? 


. Motive 4 “How. would deception benefit the 
adversary? ^7; 

s Channels — What r means of deception are 
available to the adversary? // 

= Risks — What are the risks to the adversary if 
the deception is discovered? 
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| 4), What Gan be learned from an Evaluation of 


‘Evidence (EVE)? 


* How accurate has the source's reporting 
been? 

=" Is the whole chain of evidence available? 

* Does critical evidence check out? 

* Does any evidence conflict or corroborate? 

* Is the absence of evidence unusual? 


Strengths 

Deception Detection adds rigour to analysis and 
reinforces the effectiveness of other analytic 
techniques. 


Weaknesses 


Deception is difficult to detect because it takes 
advantage of existing perceptions and biases. 


Because it is time consuming, analysts seldom 
check for deception, even when there is a well- 
known history of its use. 


UNCLASSIFIED 


. Structured Analytic Technique 


Decision Trees 
Purpose 


To establish chains of decisions and/or events 
which illustrate a comprehensive range of 
possible future actions. 


Description 


Decision trees are a way to chart the range of 
options for a given decision point, give 
estimates of value or probability for each option, 
and show possible outcomes. 


Usage 


. To aid our own decision making-by 
explicitly comparing options. 

. To create a heuristic model-óf;our 2⁄ 
subjects' decision making. ( 

. To map multiple.competirig Hypotheses” 
about an array of póssible actions; 


A decision tree illustrates a comprehénéive set, 
of possible decisions and the possible 


outcomes of those decisions. It can then be 
used to help assess the probability that the 
subject will follow any given sequence of 
decisions. 


A decision tree is structurally similar to critical 
path analysis and to programme evaluation and 
review technique (PERT) charts. However, 
both these techniques show only the activities 
and connections which need to be undertaken 
to complete a complex task. A timeline analysis 
as done in, ‘support of: a.criminal investigation is 
„~ | essentially adecisiontréé drawn after the fact, 

js nee P the dad taken, : 


eripi SML Mis "Décision 


shown as 
squares, 


„and circles - 


“fepresent 
uncertain .. 
outéomes: 


Good faith ey 
^" Bad faith 
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Cpoints are, ` 


Requirements 


Analysts producing a decision tree-to model a 
subject's-decision making need tó have a rich 
understanding of thé subjective and. objective 
worlds of the’subject(s), including: cultural 
norms and values/-motivés; opérating doctrine, 
operational environment; sense tc to risk, and 
other relevant factors. ; / 


Strengths 


Decision trees are simple to understand and 
interpret. A decision tree can be generated by a 
group through brain-storming, and can also be 
posted up for addition & comment over a period 
of time. Both approaches can increase the 
completeness of the set of possible options and 
consequences. 


Weaknesses 


Relies onthe accuracy of data used, 
completeness 'of-the range of options assessed, 


‘and on the validity of the qualitative probabilities 


Fyaluües;estimated for each option. 


A detailed decision tree can present the 
misleading impression that its authors have 
thought of all possible options and/or outcomes 
for the situation. In intelligence, there are often 
options available to the subjects which we did 
not imagine, just as there will be unintended 
consequences which the subjects did not 
foresee. 


References 


e Decision Tree Analysis by MindTools 


"Quantitative Approaches to Political 
Intelligence: The CIA Experience" by 
Richards J. Heuer, Jr. American Political 
Science Review, Vol. 73, No. 4 (Dec., 
1979). pp. 1215-1216. 
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— Structured Analytic 


Delphi Method 
Purpose 


To produce reliable consensus judgements from 
a large and diverse set of experts. 


Description 


The Delphi method is an iterative survey 
approach to generating consensus judgements. 
The progressive series of questionnaires is a 
structured, anonymous debate among the 
participants. 


The first set of questionnaires is sent out to the 
panel of experts who provide their individual 
judgements. 


The next set of questionnaires includes | 
anonymous summariés of thé responses to thé 
previous questionnairé;the panel members are: 
asked to consider these other ,responses/and to 
revise or maintain their own judgements 
accordingly. 


A reliable consensus of opinion usually 
develops in the second iteration, but the 
process can be repeated further if necessary. 


Sometimes the survey will reveal stable 
disagreement in the form of two or more 
factions holding incompatible opinions. 


Usage 


The Délphi method can be used in predictive 
assessment$, an. priórity Setting assessments, 


„| and in generating: eRe Se “based wa. temas 


G "ug. 
Requirements 

A/successful Delphi depends -on good survey 
design/sound analysis of the survey, results, 

and om involving respondents with valid@nd 
relevant expertise: 


Strengths 


Studies of Delphi method have shownvit to” 
produce more reliable consensus judgements 
than face-to-face meetings. 


A Delphi survey can also be more efficient and 
economical that arranging and facilitating a 
meeting / workshop of the appropriate experts. 
Weaknesses 


Errors i in survey design, survey results analysis, 


Lor. iimselecting the participants can lead to 
invalid résults. 


The précesg canbe laborious and time 


karse Haka 


"S Recommended Readings 


. Gazing ito the-Oracle: The Delphi Method 
and its Application to Social Policy and Public 
Health. Jessica Kingsley Publishers: London. 
Michael Adler and Erio Ziglio, 1996. 


= Studies in the Quality of Life: Delphi and 
Decision-Making. Lexington Books: 
Lexington, Mass. Norman Crolle Dalkey, 
1972. 


* Delphi Method. Wesley: Reading, Mass. 
Harold A. Linstone and Murray Turoff, 1975. 


UNCLASSIFIED 


Devil’s Advocate Advice for Devil’s Advocates: 


- " ` ^ y i wf 2 
Purpose Clatify. your role’ Substantive issues only? 


/ Inaccuracies? Spelling and grammar? 


To test an argument by attempting to prove it is 


flawed or fundamentally incorrect. = Be Careful of harping’ on the same issues; 'it 


creates “warning fatigue". 


= Ask questions; don’t‘make stators. “What 
would change your mind?” ^ C; ip 


Description 


People may not raise tough questions because they 
want to be a team player or lack self-confidence. 
However, as the official Devil’s Advocate (DA) they 
have the public licence and the responsibility to be 
critical. 


= Be diplomatic. "What is the vs. of: your belief? 
Is there another interpretation?” 


* Be assertive, but humble. "Could you rephrase 
that in simpler terms for me?” 


The Roman Catholic Church used to appoint an Strengths 
advocatus diaboli [Devil's Advocate] to argue 


against granting sainthood to a candidate. 


Finds flaws in a hypothesis or assessment which the 
author has missed. A paper that survives the 
process has more credibility. 


Courts of law/have used a 
structured confrontation ^ 
etween prosecütion.and 
t defence for/hundreds of? , 
years as @means of testing 
arguments.. 


q Although the approach is confrontational, it can 
pomo creativity and reduce “group think. E 


Weaknesses 


\y,-|‘Many issues don’t Have a simple `yes` or ‘no’ 
‘| ‘@nswer Both views may have merit. This approach 
'/may'impose-a false dichotomy. 


This process is best used when a choice is required 
between two alternatives. Many use it to pre-test an 
argument or submission before it is formally 
presented for approval. 


An overly aggressive DA can cause bad feelings, 
recriminations, and obstinacy, especially in an 
emotionally charged atmosphere. 


A DA can acquire a reputation as Dr. Doom; it may 
even be a Career Limiting Move. 


Use it to surface issues that might otherwise be 
ignored and ensure that issues, once raised, are 
addressed and not just glossed over. 


A weak DA may create a misleading impression that 
all sides of an issue have been considered, when in 
reality they have not. This could lead to false 

confidence. 


fc ara «hig "^ 


Whether cati Ma Colleague one-on-one or 

/», playing the role/at a meeting always ask 
permission, "I'd like tó explorëhis isSue. Do you 
nip. if lay Deyil s Advocate?” 


Related Techniques: 


e Team A/ Team B argues different (rather 
than just opposite) points of view 


The best thoice a DNA isa knowle M g&aii büt 
neutral party that can "t te, mgr tide of bias. | 


e Dialectic Inquiry goes beyond merely finding 
flaws. 


A good Devil`s "PANIS, goes) bejóhd, simply 
contradicting his or her opponent: They, challenge 
both evidence and assumptions, advancing-counter- 
arguments and alternative interpretations. 


e Adversarial Collaboration looks for areas of 
agreement. 


e Six Hats forces everyone to play DA. 
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Dialectic Inquiry 
Purpose 


To arrive at plausible alternative explanations 
through a guided discussion that challenges 
assumptions and evidence. 


Description 


Dialectic Inquiry is a method of examining and 
discussing opposing ideas in order to find the 
truth. It goes back to the ancient Asian idea that 
everything is made of opposites (yin and yang). 
The German philosopher Hegel expressed this 
in terms of fhesis, antithesis and synthesis. 


Dialectic Inquiry goes beyond merely finding 
flaws. Through constructive “discussion, it 
attempts to produce plausible alternative” 
analysis. 


Usage 


Dialectic Inquiry is often used to Critique a draft 
paper. It can be done one-on-one with the; ~~ 
author by a supervisor, peer, or a team; / 


The process guides the discussion with the goal 
of providing constructive feedback to the author. 
One version of the process has six steps: 


1. identify the thesis or main points of the paper; 
2. list all the evidence presented by the author; 


3. list the assumptions necessary to support the 
thesis or main points of the paper; 


4. list problems.with the evidence (E.G. poor 
quality ' Alternate, interpretation) and any 
unused, (but relevant'évidence that the author 

ordalag to coñsider 


| Suist' (Grab lark, with the a$&umptíons (E.G. 


weak; outofedatéy as well as missing 
assumptions, and’): 


6. in light of the issues inc Ahead re-examine 
the author s conclusions for’ weakhesses and 
flaws, propose solutions and alternatives. 


Requirements 


Dialectic Inquiry is a form of adversarial 
collaboration, like the Nosenko Approach. 


= Yuriy Nosenkó' was a‘Soviet jte 
officer who defected to the-US in 1964. 
Some thought he was‘ legitimate: others 
thought he was a plant. Both,sidés-were 
asked to list the evidence critiCábto their 
argument. Each side then addressed the 
composite list of 14 items in what became 
known as the “Wise Men” Report. The 
process led to a stronger belief that 
Nosenko was legitimate. 


| As such, analysts need to view the process of 


Dialectic Inquiry as a partnership, not a contest. 


Results are much better with a team. 


| Strengths 


This technique counters mindsets and bias 
through the critical, but collegial, examination of 


.| existing analytic positions. 


Thé focus is/on the argument, not the author. It 
depersonalizes the discussion. 


Weaknesses 


The process can be time consuming. To 
expedite the process, focus on the most 
substantive issues, E.G. the most diagnostic 
evidence and the key assumptions. 


The technique stresses critical examination 
over group harmony. 


Related Techniques 


Key Assumptions Check focuses on 
assumptions. 

ACH and Nosenko Approach focus c on 
evidence. 
Argument Mapping exposes the 
structure of an argument. 


Structured Analytic Technique 
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Force Field Analysis 
Purpose 


To highlight forces or key drivers influencing an 
issue or change. 


Description 


Force Field Analysis investigates the balance of 
forces acting on issue or influencing a change. It 
attempts to identify the most important players 
and target-groups involved in the issue and their 
relative influence. 


For a situation in flux such as a political crisis: 


e Describe the existing situation, 
e List all of the forces for change; and. ` 
e List all of the forces resisting change. 


To simplify matters, think of forces ‘that make 
matters either worse or better. 


Forces š 
resisting | 
change 


Eiin š 


Forces 
for 
change 


Scores or weights can be assigned to each force 
showing their relative strengths. This can be 
represented: by thicker or thinner arrows. 


k: Usage, 


if a Sas ‘contemplated, Force Field 
Analysis“ can, help/identity potential opponents. 
and allies. At thiéconSeétof. a study, it clarifies key 
drivers or forces. It also.helps\challenge 
assumptions about these drivers and highlights 
those that are stabilising or destabilising: 
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It prompts discussion over the relative impact 


generate alternative scenarios and possibilities. 


Requirements 


The “exam question” must be clearly identified 
and, understood by the study team-before 
embarking onthe identification of positive and 
negative forces. . 


Strengths 

Force Field Analysis highlights core forces / 
drivers that influence or impact a situation. If 
presented visually, it is easy to see which 


factors merit attention. This helps determine the 
best way to influence a situation or outcome. 


different forces have on a given situation and 
may expose assumptions. 

Speculating on changes to drivers helps 

ftis, useful for generating signposts and 
indicatots.."/f this changes, we should see X." 


it isa precursor to Cross Impact Analysis. 


Weaknesses 


Analysts must have a good understanding of 
the relevant drivers and forces. 


The weightings given to drivers are arbitrary 
and subjective. At most, one might say that one 
factor is twice as important as another. 


The technique does not generate a specific 
"baseline" scenario. It does not predict what 
equilibrium might look like or when it might be 
achieve. 


The "sustainability" of a driver is not addressed. 


Not all forces are clearly negative or positive. 
Gray areas may not be adequately examined. 


Further Reading 
e A Force Field Analysis Worksheet and instructions are 
available at htto:/Avww.mindtools.com/rs/ForceField. 
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Structured Analytic Technique 


Software:to create Gantt charts is abundant. All 
have similar features (see product 
Comparison). The dominant product is 
Microsoft Project (videos, blog). Two free 
alternatives are-ProjectLibre and GanntProiect. 
To create a chart from Excel, dy the Vertex42 


Template. 


Gantt Chart 


Purpose 


To visually depict the phases of a project. 


Description 


Charts can also be created Silaboratively and 
online (Creatly has a free Simple Gantt Chart 
Tool). 


Gantt charts are a process map that show how 
phases of a project overlap and where one 
phase must finish before another can start. They 
create a work breakdown structure showing 
dependencies and milestones. They help identify 
when resources will be needed and measure 
progress toward the final goal. 


Strengths 


Gantt charts provide managers with a view of a 
project’s status. 


Usage 


The can monitor the development of a new 
weapons system, preparations for military 
action, or the stages of self-radicalization. 


Analysts could use a Gantt chart to show how. 
they intended to research and write a-very 
complex paper. This one,shows the percentage 
of each task completed as of Maren 48th. 


They help.identify when activities must start in 
fader fóra project to end on time, enabling 
better planning; 


E TT TW Mar 2033 | [11 Mar2013 ^ Bet Sc SEET 
| Task Name k— TERCA 
| iM res wit areas aie Sy 


Í Preliminary Investigation aane 
I COND Sa \ 

l 
! outine 
I 


A Garitt chart can identify and describe the 

> A moduscoperandi of a criminal or terrorist group, 
including the preparatory steps they typically 
take before committing a crime or attack. 


! Collection Plan 
i 


| Research 


i 
| write 


I 
Peer Review 


Weaknesses 


When modeling the expected behavior of an 
adversary, there is always the danger of mirror 
imaging (expecting them to act like you). Your 
Gantt chart may not look like their Gantt chart. 
Although useful and valuable for small projects 
that fit on a single sheet or screen, Gantt charts 
can become quite unwieldy for projects with 
more than about 30 activities. 


A Gantt chart could also depict the steps in 
bomb-making, planning an attack, or uranium 
enrichment. It would provide a best (or worst) 
case scenario for time until completion. If the 
resources needed for each stage were known 
and distinct; ‘analysts‘could use this to argue the 
A pee had reached certain point. 


: Reguirements 


Gantt charts only convey time; they don't covey 
cost or difficulty. For example, Gantt charts 
depict work as constant. In practice, the 
workload (or effort) can vary during the task, 
e.g. it may be front-end or back-end loaded. 

A large number of dependencies may result in 
a cluttered or unreadable chart. 


The basic steps t to ‘resting a Gantt chart are: 


s Create a Task List OF all planned activities. 
= For each task, estimate the/time-ifwill take. 
= Plot each task (using software or graph ..- 
paper), showing whether the tasks/activities 
are parallel or sequential. 
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Structured Analytic Technique 


Requirements 


Heat Map 


Heat maps can be created manually by shading 
cells of a table in MSWord or Excel (How to 


Create a Heatmap in Excel). Ideally./however, 
software i$ used: É 


Purpose 


To visually highlight or group items of interest. 


Description » Panopticon ($) | 


Similar to 
thermographic 
images from 
infrared cameras, 
heat maps depict 
hot-spots of activity 
using a colour 
scale. 


= R Statistics (free) for statistical'corfiputing, 
but contains several functions to trace heat 
maps. 


» Heat Map Explorer ($) 


= Heatmap Builder (free) is a standalone 
application in Visual C£ using the .NET 
environment. 


Usage 


= iMapBuilder ($99) is a Windows map 
software program that generates interactive 
‘feat maps in flash format. 


Heat maps communicate threat levéls clearly, 
and simply. They can’ also bé:'used to show ,, ` 
whether elements of à collection plan jave been 
fulfilled. 


E Goódle Docs (free) has a heat map gadget. 


/; 4», Visual Fusion ($) creates interactive, visual 
, applications utilizing SharePoint (video). 


A simple heat map translates statistical data into-, 
an easy-to-use graphic that helps users see 
things that aren't apparent otherwise. 


Stréngths 


More complex heat maps use colour and size to 
visualize complex data values and relationships. 
An area-based visualization like the Newsmap 

below is called a tree map. 


Provides quicker analysis, better decisions and 
more effective communication. 


Rapidly identifies trouble spots, before they are 
out of control. 


Ensures that attention and resources go where 
they bring the best return. 


Discovers underlying trends that point to high- 
value opportunities. 


Weaknesses 


Depends upon the completeness, currency, 
consistency, and accuracy of source data. 


It groups s li Sone together "e. Elles them 
by topic (sports, business,’ riational; etc.) then 
uses size to indicate how many news stories 
there are on this topic at any time: \Users,can re- 
configure the data and color gradient. 


Forces categorizations and groupings which 
may lack nuance (e.g. "pixelization"). 
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- Develop one or;more plausible pathways for this 
unlikely event to unfold. 


High Impact — Low Probability 


-'Generate a list of indicators or Spiiposts for these 
alternative pathways. Í 


Purpose 


- Identify factors that could deflect this bad outcome 
or encourage a positive one. , _ 


To provide early warning that a seemingly 
unlikely event, that would have major 
consequences, might actually occur. 


Review and update the above-as necessary. 


Strengths 


Description 


This approach counters |.. 
the natural tendency to |` 
focus attention on only 
the most probable 

scenarios. 


The conventional view may be that an adver- 
sary cannot militarize anthrax because it is too 
complicated. High Impact — Low Probability 
analysis would question this view. It is a 
“contrarian technique". 


What about a low 
probability event? 


It gives analysts licence to explore the 
consequences of an event — particularly one 
deemed not likely by conventional wisdom — 
without having to challenge the main-line 
judgement or to argue with others about how 
š likely án eyent is to happen. 


It typically begins by identifying a potential event 
then attempting to put the impact. of such an 
event in context. If this reveals the 
consequences of the event are severe, this 
triggers further examination óf;«how such an 
event might come about, possibly revealing that: 
it is not as unlikely as first tunt Mapping Out the course of an unlikely, yet 

2| plausiblé\event Gan uncover hidden relation- 
An adversary, for example, may devise'an , S/a} ships between key factors and assumptions; it 
alternative means for militarizing anthrax. Théy- ..| also.carivalert analysts to oversights in the 
may improvise, skip steps, and take more risks, - mainstream analytic line. Lastly it could reveal 
ending up with a crude, but effective product. signposts to monitor for a shift in the situation. 


Weaknesses 


Usage 


High Impact — Low Probability analysis is 
advisable when analysts and policy makers are 
convinced that an event is unlikely but have not 
given much thought to the consequences of its 
occurrence. In essence, it is a warning about 
an Apreta but not impossible event. 


The evidence for low probability events is 
usually difficult to find and ambiguous. 


Not everyone has the requisite imagination. 


Convincing decision makers that they should 
consider a remote possibility for which there is 
currently weak evidence requires: 


Requirements: 


- an authoritative or credible author; 


à The steps iqvolved in this Ype of analysis are: | i f 
- plausible arguments with compelling 


scenarios and 


“Cady. desove; th e ünlikely event! 


Define the Outéome of the event (consider primary and 
secondary impacts). This, willl SHY examining what 
others see as unlikely. ^ 


- an open-minded audience receptive to 
unconventional ideas. 


If the low-probability event is several years in 
the future, don't expect anyone to take 

immediate action. Hope instead for permission 
to monitor the issue. 


- Identify indicators that the event ARa à 


- Postulate triggers for the event. What catalysts would 
accelerate things? 
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Structured Analytic Technique 


Requirements 


indicators and Warning (I&W) 


This technique works-best whenanalysts have 
good knowlédge of the process that is typically 
followed, E.G-the Steps in launching & missile. 


Purpose 


Helps track events, monitor targets, spot 
emerging trends, and warn of change. 


The indicators and signposts: must be 
observable. 


Description 


An analyst or team creates a list of indicators or | Strengths 


signposts of events that could be expected to be 
observed if a certain situation were to develop. 


Anticipating warning signs makes analysts 
quicker to recognize emerging threats and 
opportunities. 


Economists, for example, monitor “leading 
indicators” to know if the economy is improving. 


Indicators instill rigor into the analytic process 
by providing a basis for judgements. 


This list of indicators could be based on a review 
of similar events in the past or‘it,could-be 
entirely hypothetical (“If this were to happen; we 
would expect to see....”). l 


Listing indicators makes the analytic process 
more transparent and gives readers a better 
'Sénse of how far something has progressed. 


Usage zh 
Weaknesses 


I&W typically begins with a coricern over 
possible future event, say North Koreà launching 
a missile. Next, analysts break-down all the 
steps that would be required prior to the event. 
Then they would identify those steps that might 
generate observable phenomena. Lastly, they 
would task collection assets to monitor for these 
phenomena. 


Analysts can become fixated on watching for 
^the identified indicators, missing signals or 
developments that are not anticipated. The 
Yom Kippur/ War of 1973 is a good example. 
Israel did'riot seeswhat it expected, hence was 
surprised when Arab forces attacked. 


The best indicators are often very expensive or 
difficult to observe. 


In a more complicated situation, I&W could help 
determine which one of several possible futures 
is emerging. This involves: 


The adversary can use Denial and Deception 
(D&D) to obfuscate and confuse. 


° identifying @set of competing scenarios 
(possibly: via ar SAT such as Alternative 
F utures Analysis); i 


I&W does not work well on an adversary that 
does not follow protocol or doctrine, e.g. 
insurgents. 


(ereating, separate lists of potential activities, 
"g statéments), ior events (signposts) hae 
for: each, scenarió;, / 


Indicators need to be validated and periodically 
updated (see SAT Indicators Validator). 


° watching for these signposts; and 


Indicators can be misapplied to areas or 
situations where they are less reliable, E.G. US 
indicators may not apply in Canada. 


e identifying a specific ic developing Scenario, 
based the signposts observed. 
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Structured Analytic Technique 


Which indicator is,best for determining if 
someone is concealing.criminal proceeds? 


indicators Validator 


Purpose 


Hiding 
Nyy | / money from 
.Crimifiat, ^ business Distrust of 
‘activity partners 7 / „banks 

* Makes large HL ` V Ñ HU 


cash deposits 


To determine the strength (reliability) of an 
indicator. 


Description 


Sends money HL HL Z, /HU 


Not all indicators are created equal; some are oroad 


better than others. The best indicators are 
consistently present when and only when the 
anticipated event is about to take place. This 
technique orders a set of indicators according to 
their value in reliably preceding an event or 
highlighting a situation, person, or issue of 
interest. 


Ë Makes unusual 
== deposits and L G U 
(ey. withdrawals 


3 Æ Moves money in 
== and out of L C U 
= account rapidly 


Refuses to 
2 answer questions 


Usage 


The best indicator would be HL in the first 
| column; and HU in all others. In this case, "large 
Cash deposits" is the best choice. 


Monitoring the wrong-indicators can be expens: 
sive, time consuming.andlead-to “false. ` 
positives”. Whether the indicators relate to 
possible money laundering or'an, enemy S 
intentions to attack, this technique helps. focus - 
attention and rationalize efforts. 


Presséd.for time? Don't bother with multiple 
scenarios, assemble a list of all known 
‘indicators for the activity in question, determine 
how: frequently each indicator has been 
obsérved; and rank order the indicators by 
frequency ; 


Requirements 


Begin with simple grid or matrix. Down the side, 
in rows, list the set of indicators currently used 
to detect or monitor a given activity or anticipate 
an event. Across the top, in columns, list the 
activity or event of interest followed by similar or 
related activities or events. 


Strengths 


Focuses attention and rationalizes efforts. 


Weaknesses 


Now, (assess for each cell of the matrix, whether 
the indicator,is: ^. ) 
= Highly likely; to appear (HL) 
hb "^. Likely to appear (t) 

£v... Could.appear (C)... 
JUs “Unlikely to, appear (Üy 

š Highly Unley to appear (BU) 


This technique can only identify the best 
indicator(s) in a given set. If these indicators 

were developed from relatively few instances or 
events, their reliability is inherently suspect. 


This technique does not generate indicators. 


Indicators (even validated ones) become less 
reliable as they move further away (temporally 
and geographically) from the activity or events 
upon which they are based. 


The most ‘discrimi inating’ indicator. is Highly 
Likely to emerge in one scenaric: and Highly 
Unlikely to emerge in any otHef scenario, The 
least discriminating indicator is Likely‘or-Highly. 
Likely to appear in all scenarios. 


Source: Based on indicators Validator, Pherson and 
Associates LLC. 
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Structured Analytic, Technique 


issue Redefinition Usage 


Issue redefinition is worthwhile whenever the 
Stakes‘are high-and mistakes are expensive. 


Purpose 


To clarify the needs of a client. à 
Requirements 


Description 


It may be productive to PATI the problem. 
(see Taxonomy of Problem Types). You could 

also think about whether the questiónvis closed 
or open, set in the present or the future. 


"It's a nice paper, but not what | needed." 


These words signal a disconnect between 
analyst and client. Was the client unclear or did 
the analyst misunderstand the tasking? 


e Closed questions have a definite answer. 
e Open questions require going beyond the 
data. 


To avoid this problem, analysts should interview 
their client, exploring the following six areas. 


Present 
Can be answered 
by “yes” or “no”. 


The answer depends on 
understanding how an 
individual, organization or 


The Question—What does the client really want 
to know? What answer would satisfy’them?Vhy 


is this important to them? How will they use your / E.G. Does Syria system is likely to behave. 
analysis? How does the;client respond when have a CBW 
you: program? E.G. How much Sarin will 


$ Syria have by 2020? 
‘There are arange of | These are questions 
possible about future hypothetical 
explanations. possibilities. 


Broaden the focus 
Narrow the focus 
Redirect the focus 


EG; Why does E.G. How would Syria 


Turn things 180 degrees Syria feel it needs react to US pressure to 
. , CBW?) destroy its CBW 
Level of Analysis—How much does the client f stockpiles? 


want to read, a two-page overview or a 40-page Source: UK PHIA Analyst Guidance Note, 2011 


in-depth study? Raw, summarized, or 
assessed? 


Strengths 


Saves time and money. 


Time Frame—ls a quick (80 percent) answer in 
three days more important than a full answer in 
two all 


Provides an opportunity to educate your client 
on the depth and range of your capabilities or 
alert them that the question is outside your area 
of expertise. 


inion “SdurcesDoes the client have any 
>; „Suggestions of preferences? What does the 
$ dlient, think of yaur ideas? 


Weaknesses 


Pressat R Eleirieits—Should the final\product 
include statistics, chars, mages. appendices,’ 
etc.? 


Clients often don’t have the time or inclination to 
answer all or an analyst's questions. 


Anxious to please and under tight deadlines, 
analysts may curtail this process and 
commence researching and writing 
prematurely. 


Delivery Mechanism—Does thé: client want a 
hard copy or an electronic file? A verbal 
briefing? PowerPoint? 
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Taxonomy of Problem Types 


Characteristics cl Problem. Types x C E 


Deterministic Moderately . Severely " ""ndetérminate 
Random <| Random | . Cus OE 


What is the Obtain How much? Identify and Identify ‘Predict future 4 
question? information | How many? rank all outcomes | events/situations ` 
outcomes in ON a, ING 
unbounded 
situation 


Analytical task Find Find/create Generate all Define Define futures 
information formula outcomes potential factors 
r outcomes 


Analytical method | Search Match qata to << | Decision Role Analyze models 
sources formula “theory; utility playing and scenarios 
BA. Es analysis ~.. and 
UA LT T gaming 


Analytical Matching ,| Mathematical Influence,. ~ 7|Subjective | Use of experts 
instrument -| formula diagram, Y 4). evaluation 

|, utility, — of 

“probability>, | outcomes 


Analytic output Fact Specific value Weighted af. D Plausible Elaboration on 
or number altemativ& s, | outcomes | expected future 
outcomes , LS 
Probability of Lowest Dependent on | High to Highest 
error data quality very high 


Follow-up task Monitor for Repeated Exhaustive 
change testing to learning 
determine 
true state 


id Source: Joint Military Intelligence College, 1991 
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Structured Analytic Technique 


= Are the assumptions, consistent with each 
other? If they are not, is there a reasonable 
explanation? 


Key Assumptions Check 


Purpose 
= How much confidence exists, that,this 


To expose and validate the key assumptions on assumptions correct? Woul iC3he($ ágree? 


which fundamental judgements rest. 

a What explains the dégree of confidence in 
this assumption? Is‘the, sitüation familiar or 
well-studied? Is the evidence pempglling? Is 
just “common sense”? 


Description 


Identifying hidden assumptions can be one of 
the most difficult challenges an analyst faces, 
as they are ideas held — often unconsciously — 
to be true and, therefore, are seldom examined 
and almost never challenged. 


Assess 
a Which assumptions matter most? 
= Ask “If this assumption is wrong, what impact 
does it have on the analysis?” 


A Key Assumptions Check is useful at any point 
in the analytic process. If done early, it helps 
identify collection gaps. Midway, it may 
generate alternative hypothesésAt the-end, it 
exposes weaknesses in key judgements. 


Revise 
» Re-assess and revise assumptions as new 
information arrives. 


It is highly likely that this process will identify 
}-factors.that need follow-up. Analysts may also 
décide to’explicitly state key assumptions to 
readers ‘alerting them to limitations in the 
analysis: 


Usage 


First, the analyst writes down their theory-of he 
case or hypothesis. "/ believe . 


Next, the analyst provides supporting evidence: 


Requirements 
“My belief is based on...” d 2. S yr. 


A wiflingness'to,probe the basis of (cherished 
and long-held) beliefs. 


Finally, the analyst goes through a 5-step 
process to examine key assumptions. This is 
best done in a group. 


Strengths 


Identify 
s What must be assumed to be true for the 
hypothesis to be true? 


Explicitly identifying assumptions helps: 


« Explain the logic of the analytic argument 
and expose faulty logic. 


Limit ` ' 
. Ae cil&ümstánices forcing too many 
assumptions?“ 

‘os, Isthere a way’to jéduce c or eliminate any 
Sr. ) + y ON tiens? 


» Understand the key factors that shape an 
issue. Stimulate thinking about an issue. 


a Uncover hidden relationships and links 
between key factors. 


= Identify developments that would cause an 
assumption to be abandoned. 


Check ee i 
. Why "must eai aes nbtion be true? 
= Could this assumption Have been true in 
the past, but not now?“ 
a Are there conditions or circürstances 
when these assumptions might not hold? 


= Prepare analysts for changing circumstances 
that could surprise them. 


Weaknesses 


It is difficult to know when to stop. 
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Poor Assumptions: The Root of Most Intelligence Failures 


An assumption is something that is taken for granted (without proof); We make hundreds of 
assumptions every day. They simplify our world, freeing our mind to focus on other matters. 


On occasion, however, poor assumptions have had disastrous Sons =) Wa ya 


e |n 1963, the CIA was watching Cuba very carefully. They aei thatthe Soviets would 
never risk such a provocative move as placing missiles so close to the US ' 


e |n 1979, the CIA assumed that the Shah of Iran, one of the world's most power 
monarchs, could not be overthrown. They were completely caught off guard by the Iranian 
revolution. 


e |n 2003, it was assumed that Saddam Hussein still had weapons of mass destruction. That 
led many analysts to misinterpret the trace evidence that they were seeing as evidence of 
an ongoing program. 


Should | alert my readers to my assumptions? 


Molvania: Situation Worsening 


Key Points 


* Capite: tgrt 
* Witesortss goremmers 


Just as you should alert readers whenever a key piece of 
evidence is suspect, you,should alert them whenever you, 
make a significant assumption (i.e-one that, if iñoorreét, 


would have significant impact,on your, conclusion). RecentEvenis | 
Jag text is merety apie noder vow f: 
thal & verzgregh would romsiiy pe ree 
This text os mentiy eolent moder soy | ` Bee 
trate pamper would somety teree | fi 
Background t 
Tsis text is memi aciem boderb show |" eem 
thats persgres® wouid momaily pe tee [| - i 
This textes mesece epiac hedero moy | Š 
toet a cep mon would sormmeily pe hee: 
Implications 
TU texts mersiy egies P moder to gnow Stat e paragnepin 


wow coma ge mere. Tris text or evet: e pace Molter 5c. 
show Tate persgrsce would omisi 2e here: 


E.G. “This paper assumes that the military wit continue to 
a the government in power. On this basis we conclude J 
that .. 


How can | alert my readers to my assumptions? 


One option is to create a box listing 3 to 5 key assumptions. 
This prominent display warns readers that the paper's Key 
Judgements could be seriously affected should these 
assumptions prove invalid. 


&ssessmertBase 


Tas repot's pese2 oc s singie numm sonce 


Another pii is to mention them in the body of the document. Here are some suggestions. 


° Oun cáieulations, are based on known factors, but assume that .. 
° We Ravesalso assumed .. 
“All of these timelines assume that .. 
‘On, the: assumption that, .- 
* These: timelines depend 6 oma a larde, number of assumptions .. 
If our assumption | iS incorrect .. 
A second sét‘of assumptions surround the .. 
We do not have a full intelligence picture therefore we assume the following .. 


Not only do these statements alert thé‘reader to problems and limitations, they show that careful 
thought lies behind the paper. 
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Structured Analytic: Technique 


Requirements 


Linchpin Analysis 


Accurate judgement by,analysts inselecting and 
défining the drivers, linchpins;and other factors. 
Collection.of-áccuráte-information relevant to the 
drivers, linchpins, and other factors. 


Purpose 


To manage substantive uncertainty when 
developing predictive assessments. 


Strengths 


Description 


Provides a reliable approach to predictive ` 
analysis. Because linchpin analysis requires 
analysts to state their assumptions about the 
factors driving upcoming events, it also provides 
a structure for review and revision of those 
assumptions and hypotheses. By considering 
alternative drivers and linchpins, analysts can 
generate a set of multiple plausible future 
developments with corresponding indicators. 


Linchpin analysis structures a predictive 
problem through defining the factors which 
drive and influence the outcome of a situation. 


These factors are: 


s Drivers - key variables and unknown factors 
which are most likely to determine the 
outcome of a complex situation 


s Linchpins - working assumptions which: »~ 
connect the drivers; there's usually greater 
uncertainty about linchpins thán about 
drivers 


|MWeaknesses 


Errors in’selecting and defining the analytic 
factors Willlead to flawed forecasting. This 
stage in the analysis is particularly vulnerable to 
.| cognitive biases such as mirror imaging. 


s Triggers - plausible developments which 
could Tanpi the linchpins holding the 
argument together 


| Recommended Readings 


s Signposts - indicators that the judgement 
needs revision. 


s Articulation.of Assumptions, Note 3 in A 
Compendium of Analytic Tradecraft Notes, 


Central Intelligence Agency, Directorate of 
Intelligence. Jack Davis, 1997. 


Linchpin Analysis was adopted in the CIA under 
Deputy Director of Intelligence, Doug 
MacEachin, in the early 1990s. He wanted 
structured argumentation that used non- 
academic terms so "key variables" became 
drivers-and "hypotheses" concerning drivers 
became linehpins:O ( 


= improving CIA Analytic Performance: 
Strategic Warning, The Sherman Kent 
Center for Intelligence Analysis, Occasional 
Papers: Volume 1, Number 1. Jack Davis, 
2002. 


= Psychology of Intelligence Analysis, Center 
for the Study of Intelligence, Central 


Intelligence Agency. Richards J. Heuer, Jr. 
1999. 


a, RaR, 


uapa A T is used i in forecasting / 
predictive BOPP, !] 
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Link Diagrams 
Purpose 
To graphically depict relationship data. 


Description 


Link diagrams aid investigators and analysts by 


uncovering, interpreting, and displaying 


complex information in easily-understood chart 


form. They show the associations (“links”) 


Home Health Care Fraud 


>> | Strengths 


Consider using a link chart whenever it is 

important to: 
e integrate data from different sources; 
e highlight key relationships; 


e provide a visual briefing aid for team 
“members, prosecutors and juries; or 


° show When. cases salen 


Y Requitements "a 


For ‘Clarity. dne ederet interpretation most 
charts: | V 


as circles; 


* place individuals within their 
organizations; 


Structured Anal 


between people and organizations (“entities”). 


* Ft 
> Vm f 


: LN ante. are a concise way to tell a simple 


show segreta as (Pures ‘arid, people 


tic Technique 


= linkindividuals and organizations with lines 
(solid for Confirmed links, dotted or dashed 
for Suspected Tinks) 


= suse coloured links to represent phone calls, 
financial transactions, sammomiy flows, 
etc. 


avoid crossing lineis; 
* place key entities in the ceritre; and, _ 


=a include a legend or key especially if the 
chart will be used by people who are 
unfamiliar with such products. 


Some analysts prefer to create an Association 
Matrix, prior to creating a link chart. 


Software such as Analysts Notebook 
($) or Sentinel Visualizer ($) feature the 
use of "icons", the ability to make 
changes easily, and to embed extra 
? information. REFlow ($) is a cheaper 
bpt less capable program. 


story: put éven with massive cases involving 
hundreds or thoüsands of entities they can help 
analysts find key relationships. 


A link diagram is an excellent tool for examining 
call patterns and financial transactions. 


Some software feature Social Network Analysis 
(SNA) metrics such as "degree centrality" for 
finding key players. 


Weaknesses 


Special care must be taken not to overstate 
relationships. Drawing a solid line between two 
people or between a person and an organization 
implies a confirmed connection. 


Also, analysts should interpret SNA metrics with 
caution. An individual may appear to be very 
central, but the dataset may not be complete, 
skewing the results. 
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Structured Analytic Techniques ` 


Mind Maps 


Purpose 


To improve critical and creative thinking by 


depicting relationships visually. 
Description 


Analysts often start collecting information 
without sufficient consideration of the problem or 
question they are facing. Whether what is 
needed is a conceptual framework or a 
collection plan, a mind map can better illuminate 
the path ahead. 


How to Create a Mind Map 
Adapted from Head First’ by, Tony Buzan 


. Write your topic in the centre of a large, ` 
piece of paper. 


. From the central topic, radiate out key 
words and the most important ideas you , 
have about the topic, each ona: 
separate, thick line. 


. Branch thinner lines off the ends of the 
appropriate main lines, to Show 
supporting data (the more important the 
data, the closer it should be to the central 
topic or idea). 


. Use images and colors freely in your own 
special code to show people, topics, 
themes, associations or dates, and to 
make the Mind Map more memorable. 


For examples) see; sample mind-maps. 


CC... 5 £2; 
.|Usage- 
C] Uu 

° ^ 9); / 


Mind: maps TNI: Suited to o linim 
because they. dori'timpose , linear thought (likeva 
hierarchical "table of, contents" approach does). 
One thought often spawns another, sometimes 
connecting things in unexpectéd-ways: 
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@ ithe 
Requirements 


Mind maps can be created by hand using pencil 
and paper, onthe web (Comapping, Mindomo Mindomo, 
MindMeistér/and- bübbl.us), or with Software 
(ConceptDraw, FreeMind, dopicScape, - 
iMindMap, PersonalBrain, or MindManager ). 


Strengths 


Mind maps are simple to create. They are 
intuitive, reflecting the way you think. 


They visually capture, organize and 
communicate ideas and information effectively. 


Users have greater retention and recall over 
traditional text-based approaches. 


| They provide a single view of all the aspects of 
| aproject. Lots of information can be taken in at 


a'/glance; 


Weaknesses’ 


A Mind maps are not usually appropriate for 
formal Bjreentetions, 


Recomended Readings 


Web 2.0 and the Intelligence Community 
(blog) 


OneBigWeb (blog) 

Mind Mapping Software (blog) 
Mind-Mapping.org (blog) 

Mind map (Wikipedia) 

A primer on mind mapping (blog) 
Concept maps (Wikipedia) 


eee eee 
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Structured Analytic Technique 


Outside-in Thinking Strengths 


Determining, how external changes/forces 
might, over time; profoundly-affecta, 
situatiori/issué; provides a wider cóncéptüal 
and contextual. framework. This may uncover 
additional factors, amimportant dynamic, or a 
relevant alternative hypothesis: ^5. 


Purpose 


To identify the effect of external forces, factors, 
and trends that indirectly shape an issue. 


Description 


Outside-In Thinking reduces the risk öf missing 
important variables (especially if the process is 
guided by a mnemonic like STEEP or SIPOC). 


Analysis involves understanding the interaction 
of variables, usually within a simplified model of 
reality. Some variables are *endogenous"; they 
are affected by other variables within a model. 
Other variables are “exogenous”; they are 
influenced by external forces. Understanding the 
impact of external forces on these endogenous 
variables is the goal of Outside-In Thinking. 


Weaknesses 


At the outset, it is difficult to know how much 
impact external events/factors might have so it 
is hard to gage how much effort is justifiable. 

The total cumulative impact could be negligible. 


This technique provokes the analyst. to-consider _ 

the broader context; countering the natural z, ^^ la, 

tendency to zoom-in rather than zoom-out© ;,.- ^; Théenumber of variables could become 
WO, * ) unmanageable. Casting the net broadly also 

means shifting through a lot of irrelevant 

a| materialldt;Carybe difficult to determine what to 

5 -Keep and what to throw away. 


Usage 


This technique works best at the beginning, of a 
project as it helps to identify all the critical, 
external factors that need to be researched. 


This is &diffícult technique without a team since 
it requires prong jpowisdos 


Requirements 


Example: House Prices x 


To list the key external factors that could have 
an impact on the topic/question, analysts need 
to think broadly and creatively. 


Let's say you were analyzing house prices. 
Key variables might be location, number of. 
bedrooms, age, etc. — u ; 


e For geo-political matters, consider Social, 
Technical, Economic, Environmental, and 
Political factors Er STEEP model). 


While all these variables aun direct. pad 
there are other variables such as population, 

interest rates, and demographics that have. : 
indirect impact. ` 


be For busihéss processes consider Suppliers, 

-|-7 c Anpáts,. Process; ‘Quiputs, and Customers 

Oy} (SIPOC) on Strengths; Weaknesses, 
Sppodies, a and Threats (SWOT), 


If more people move into the city the | 
demand for housing goes R followed by 
house prices. 


Next, analysts assess. s Ry each of these factors 
could affect the topic. ~& 


Tho. increase in demand is s the “rising ide x 
that floats all boats”. This exogenous variable 
could be the most importantfactorin — 
determining whether house puces; are gomg 
to rise or not. o 


Finally, they research the impact these. fattors 
actually have on the particular issue. 
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Structured Analytic Technique 


RESEARCH AND PLANNING; Subject determines how 
to execute the attack. Typical signs include information 
gathering; target reséarch, Suspicious inquires, and 
target surveillance. Those who engage in “stealth 
behaviors" are morë likely to commit violence than those 
who complain (ding attention to themselves), 


Path to Violence 


Purpose 


To assess the threats and risks posed to public 

order by groups and organized crime and to help 
guide operational decision making and resource 
allocation. 


PREPARATION: Subject acquires weapons, training, 
transportation and equipment. Choóses Clothing 

(disguise). Performs religious rituals. Writes to, Sté 
members. Shoots martyr video. 


Description 


BREACH: Subject tests security. Conducts ae run. 


The “Path to Violence” model by Calhoun and 
Weston (2003) postulates an ascending scale of 
six observable steps — the “continuum of intent” - 
which ultimately culminates in violent activity. 
Individuals or groups may move up or down the 
ladder - toward or away from taking direct 
action. 


DIRECT ACTION: Subject commits violent attack. 


Strengths 


Communicates threat levels clearly and simply. 


Focuses on observable actions and behaviours 
to provide good insight into motive and intent, 
avoiding contentious issues related to psycho- 
"l'social;religious, race or ethnicity-based 
profiling. assumptions and focuses on criminal- 
related behaviors. 


|, Weaknesses 


i RESEARCH 
& 


i PLANNING 


Dépendé'upon the completeness, currency and 
accuracy/of.information. 


PATH TO 
CRIMINAL ACTION 


| Variations over time and between individuals 
poeticum l — and groups introduce uncertainty. 


vange The model does not predict how quickly the 


subject will pass from one step to the next. 


This methodology can be used to asses an 
individual or group’s propensity to commit violent 
acts. Jt was used by the Joint Intelligence Group 
during the 2010 @lympic Games in Vancouver to 
anticipate eriminal Saas Se 


Indicators (for each step) can be subject to 
interpretation and may be ambiguous. 


References 


* Contemporary Threat Management - A Practical 
Guide for Identifying, Assessing, and Managing 
Individuals of Violent Intent, Calhoun & Weston, 2003. 


Study course 


K. Requirements 
 Aséoss. the stage of the individual or. EIVA 


GRIEVANCE: Suibject Sbi anger, jealousy, sense ot 
injustice or loss and à désire for! purpose status, revenge 
or recognition. This is usually: a. result of Being aggravated 
or wronged in some way. 


=a The Path to Terrorist Violence: A Threat Assessment 


Model for Radical Groups at Risk of Escalation to Acts 
of Terrorism, Dean T. Olson, 2005. 


IDEATION: Subject considers violence (including.criminal 
action) a justifiable option. 


s Threat Assessment and Management Strategies: 
identifying the Howlers and Hunters by Calhoun & 
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Path to Violence: Case Study 


The Vancouver 2010 Joint Intelligence Group 
(JIG) used Calhoun and Weston's (2003) 
framework, The Path to Violence, as a predictive 
assessment tool to identify radical or criminal 
groups at risk of escalation to acts of violence. 


To meet the unique needs of the JIG, some 
modifications were made to the framework. 


- Stage Indicators were developed relevant to 
protest groups. At the “Ideation” stage, for 
example, the fact that a group is considering 
criminal action as an option may be evident in 
their discussions, Internet postings, media 
releases, statements, and propaganda. 


- Each of the six stages was: colour-coded. 
This clearly communicated to JIG.clients 
where a group was within the continuum, of 
intent. 


Results 


The Path the Violence was found to be well- 
suited to assessments and situation reports: By 
identifying precursor activities indicative of plots 
to commit criminal acts, it permitted the JIG to 
assess radical groups and individuals and gave 
authorities the opportunity to be proactive and 
disrupt conspiracies. 


However, the methodology is not without its 
weaknesses. The risk posed by a group can 
easily change depending on their mood and 
circumstances. The JIG found that many groups 
and individuals remained in flux between the 
Grievancé’and, Ideation, stages where others 
reached the ‘Breach stage-only to return to the 
"Research and Planning stage. 


Groups can.also stop their advancement along 
the Path‘for-a number of. reasons such as the 
loss of critical mass necessary to commit the 
crime, alienation of allied groups, arrest of key 
members, or by pressure from law-enforcement 
efforts. 


The methodology acknowledges that a group / 
perpetrator can move in either direction along 
the path. It also acknowledges that traversing 
the path may take months, even years, or it can 
be covered in. hours, even.minutes. This is an 
area that requires further research to/be of 
greater use to law. enforcement. 


CASE STUDY 


Group X is an umbrella organization, uhiting 
representatives from various activist groups to 
mobilize against the Games. It operates as a 
leaderless collective, meeting regularly to 
discuss resistance efforts and encourages 
other anti-Olympic organizations to participate 
in its movement. The group has participated in 
workshops against Olympic sponsors. Criminal 
acts - including assault and vandalism - have 
been committed by group members. 


Analysis 


Group X is at the IDEATION stage. While 
criminalacts have been committed, these were 
neither planned nor endorsed by the collective. 


Group ‘is now collaborating with other activist 
groups, both nationally and internationally. 
They are.also'participating in local events to 
garner support,and funding for their cause. 
Plans to disrupt the Games are advancing. 


Analysis 


Group X is now at the RESEARCH AND 
PLANNING stage. They are expanding their 
network and influence. 


Group X is now the dominant organizing body 
for anti-Olympic resistance. They encourage 
protest and endorse criminal direct action 
tactics. They have attempted to bypass security 
at an Olympic venue and committed acts of 
vandalism against local businesses and 
Olympic Sponsors. 


Analysis 


Group X is at the DIRECT ACTION stage. 
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Peer Review 
Purpose 


To solicit constructive criticism and determine a 
paper's suitability for publication. 


Description 


Peer review involves getting verbal or written 
feedback from colleagues. The process can be 
formal or informal. 


Usage 


Analysts circulating a paper for peer review 
should inform their colleagues of the purpose of 
the paper and the intended audience. They 
should ask for help where they feel the-paper:is 
weak. They may also wish to solicitthe views of 
non-experts as they. may-question what others 
take for granted. 


Reviewers clarify, challenge, and suggest: 
Comments should be tempered by the cultural 
context and the organizational dynamic. 


Maintain a positive, respectful relationship. 
See the positive as well as the negative. 


Make comments in the form of questions. 


*Do you mean that ..." 

*[s there another interpretation 2e 

“Are you implying that .. 

“Might. some lagoons might overlook this point?” 


Invite a collaborative discussion rather than offer 


Py advice. 
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(9, iss: are: the pros and cons of moving these 
i: ' Sections” 
= "How sure, aré we, that. 


Don't “asta the citer person with feedback. 
Ask where your efforts are most needed: 


= “How can l help? Am I checking for errors and 
omissions, argumentation, or citations and references?” 


UNCLASSIFIED 


Requirements 


A‘successful peer review process requires 
peers with the time, inclination, and/expertise to 
provide quality feedback. The author must also 
be open to new idéasyand criticism. 


A checklist or feedback form is a good way to 
guide the process. 


Is the purpose and scope of the paper made clear? 
Are there errors of fact or interpretation? 

Is all of the discussion relevant? 

What should be expanded, condensed or omitted? 
Are all statements clear or ambiguous? 

What underlying assumptions does the author have? 
Has the author been objective? 


Strengths 


The peer review process is very good for 
identifying ways to improve a paper. 


Péers, particularly those outside one’s 
organization,’ may expose issues invisible to the 
author due to his/her inherent cognitive biases. 


Weaknesses 


It may be difficult-to find peers. The topic may 
be highly specialized and the classification level 
may severely limit who can see the paper. 


Potential peers may be too busy or act 
unprofessionally. Their comments may be 
superficial or incomplete. Their aim may be to 
embarrass the author or to protect their position 
by blocking publication and curtailing debate. 


Only using friends and like-minded individuals 
as reviewers won't provide a full range of 
critical comment. This mutual backscratching 
merely reinforces existing views. 


If the paper is urgent, peer review may take too 
long, especially if it is done sequentially. 


Peer review deals poorly with innovative, 
ground-breaking work. Many important 
scientific papers were initially rejected by peer- 
reviewed journals. 
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PEER REVIEW FEEDBACK FORM 


3 NM. 

L] Partner Document 4 
Authors(s) (Specify): 
REVIEWER 


Reviewer’s Location aa Time spent on review 


Anonymity Requested 


SUMMARY EVALUATION 


2 3 4 5 | PRESENTATION), | 


HEORY. Is the study-concéptually based and theoretically grounded? 
METHODS i 


Is thé-writing appropriate. clear and unambiguous? 


—-|-— 
521. 
+ 
Un 


— 
t3[r2|t2 


—— m 
ain} ey Gr 


! 


mj 
Fr lsi 
tal} [|> 


Is this 4-substantial, original or innovative contribution? 


Are the issues dealt, within-a complete fashion? 


WIPWI] LG} WI] wjn Ww] 29] 22 
+|+ |+ |+|+ |+ |+ |+ |+ |+ 
Un 


i 


[] 
oO 
Z 


O 
)MMEN 


A 


EC 


= 
m 


NDATION 


|O 


5 = Accept publish in present form: 4 = Return for minor revisions: 

3 = Return for major revisions —please return to reviewer for 2" evaluation after modification: 

2 = Reject needs extensive revision but could be resubmitted: 1 = Reject fundamentally flawed 

aA 5 Should this paper be accepted for publication? 


: 


[] YES L] NO Is a written response from the author(s) requested? 


Sienature of Reviewer: Date: 


uted’ or hypercritical comments of marginal use: 1 = Useless review, inappropriate 
remarks 


Irrespective of the final recommendation of the review, how helpful or 
valuable were the comments in improving the report? 
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Presentation 
= Does the paper tell a cohesive story? 
= |s a tightly reasoned argument evident throughout the report? If not, 
where does the paper wander? 
= Do the title, abstract, keywords, introduction and conclusions 
accurately reflect the major points in this report? 
= |s the executive summary and abstract clear, concise and to the 
point? 
= |sthe overall length appropriate? What portions of the report should 
be expanded/condensed/combined/deleted? 
= Has the author structured the document in an appropriate fashion 
(i.e. headings, annexes, etc.) with headings acting as a logical 
framework or road map supporting the "flow" of the document? 
Are graphs, figures, photos, tables etc. clear, appropriately 
formatted, and necessary to the argument or discussion? 
Do the legends for tables, figures and charts convey exactly what 
the reader should expect? 


Style 
= |s the writing concise, easy to follow and interesting? 
= Is the writing style appropriate? 


References 

= Are all (and only) pertinent references cited correctly? If any are 
missing, please reference in the review. 

= Are references provided for all assertions of fact not supported 
directly by data in this report or those not well known to potential 
readers? 

= |s it clear how this work is related to work already accomplished 
and/or published? 


Context 

= Does the introduction clearly set the-context (scientific, military, 
project, historic, intellectual, policy, etc) im which this work was 
undertaken? 

= Does this work properly place itself in the context of previous Work? 

= |f applicable, is it clear who requested the report; why, and how it 
will be used by them? 

= |s the practical relevance of this work clearly articulated? 


Theory 
= |f applicable, is the underlying theory/mathematics clear, accurate 
and appropriately developed and complete? 
= |s other theoretical and methodological work properly referenced 
and built upon? 


Methods 

= Are the methods used appropriate? Current? Overly complex? 
Simplistic? 

= Are the methods described clearly enough that the work could be 
repeated or extended by someone else? 

= Are all underlying and necessary assumptions clearly stated, and 
the implications on the analysis clearly articulated? 

= If software tools were used, is their role clearly understood? Have 
titles and Nersion numbers been provided? 

= Are there errors’ in, inference, interpretation or mathematical 
analysis? < - 


iscussion , 

^re significant statements Ov/Agducti GH justified and based on 

)/ evidence, contained within this repott? 

= ‘Isitélear Whatii is. known: (confi Imen) and what is speculation, 

opinion; 6r conjecture? / ( 

= Does the discussion, propefly. draw out and illuminate key results? 

= Does the discussion haye a logicát-flow andre the arguments 
cohesive, clear and concise? (> 

= Are all key concepts understandable? 

= Does the conclusion synthesize the main gints of thé report, 
leaving the reader with a deeper understanding? ; 

= Can you easily verify results and conclusions by examining) the 

associated tables and figures? 


VA, 
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Recommendations 
= Are any recommendations made? Should there be? 
= Do the.recommendations offered follow from the findings of the 
analysis? 
Is the strength/tone of the recommendations appropriate? 
= ‘Are the recommendations likely to be useful to the partner? 
= Do they provide a good guidance for future research? 


Significance 


= In your expert opinion does this report make a signifi icai or novel 
contribution? 


= Does this report provide a.strong Tyra for follow on 
research? 

= Are there errors in technique, fact; aisida or interpretation? 

= Is this report scientifically or analytically rigorous; ‘accurate, and 
correct? 


Merit 

= |s this report of a type appropriate for regular publication? If not 
suggest what forum might be more appropriate? 

= Is the work novel and relevant? 

= |f this report were not published would it have any impact? If not 
then why should it be published? 

= |s it clear why this work was done, who asked for it, and how it will 
be used? 

a Does this report demonstrate thoughtful, scholarly 
formulation/implementation? 


Completeness 
= |s the report marked by a unity and continuity of parts, showing 
interdependence between these parts? 
w /|s the report self-contained? 
w ¿|s the overall balance correct for the report? 


Policy 
= Are the opinions expressed by the author scientific in nature and 
do not reflect policy judgements that are either inappropriate to the 
organization, or are inadequately supported by the data 
presented, or are not congruent with departmental policy? 


= Are issues dealing with policy carefully and objectively 
Considered? 


Security 
= |s the report properly classified? 
= Are there “grey areas" within the document, with respect to 
security issues? 
= Are distribution restrictions appropriate and consistent with polices 
and security guidance? 


Written Comments 
Reviewers are required to provide a one or two page summary 
identifying the problems found (or accomplishments observed) and 


suggesting how the report might be improved. These comments should 
expand upon the simple numeric grading used in the proforma. 


= What are the questions being addressed by or major contributions 
of the report? 


= What are the major weaknesses and strengths of this report? 
= |s this paper scientifically and technically sound? 
= |s the method or implementation novel and nontrivial? 


= Are the conclusions supported by the data, theory or arguments 
presented? 
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Structured Analytic Technique 


Pre-mortem Assessment 


What if my main-conciusion or key 


Purpose judgment turns out'to be flat wrong? 


DRM 
To identify and examine the weaknesses in a E 
A ^ * \ l st 
judgment or assessment by assuming it was evidence? / 
incorrect and asking “How did we go wrong?” ee y 
What ` 
É shouid have N 


Description t _ the absence of | 
\ evidence told / 

The Pre-Mortem Assessment is “an audit before " m. de 

the fact." It asks analysts to imagine that time Usage 

has shown that their judgement was dead 

wrong. The objective, now, is to figure out how it | Analysts should use the Pre-Mortem Assess- 


failed. ment just before finalising key analytic judge- 


ments. Teams should use it as soon as the 
If done in a team, the process begins with group starts to coalesce on a consensus 
structured brainstorming. "What Jed to.this position. 


unexpected event? What were the signs it was 


going to happen?" Requirements 


A comprehensive assessment of possible 


This technique requires a dispassionate and 
sources of analytic error should include: 


detached attitude. There can’t be any vested 


I ' : ^ interests or “sacred cows”. 
* Key Assumptions — What assumptions were 


made? Were any weak or unsubstantiated?- s| Strengths 
Should these have been highlighted? What 


was the impact of an invalid assumption? The primary góal,of the Pre-Mortem 
TS e Assessment is'to reduce the risk of surprise and 
= Credibility of Critical Evidence — What the subsequent need for a postque i 


evidence =: stream of reporting might Pai investigation by providing a totally unbounded 
been false? What impact did this have’ yet highly structured mechanism to explore all 


- Contradictory or Anomalous Evidence — Was the ways an analysis could turn out to be wrong. 


any data rejected or ignored? If so, did it 


suggest an alternative hypothesis? Many members of small groups suppress their 


dissenting opinions. The technique gives team 
members that have unspoken reservations a 
license to speak out against the group 
consensus. Because identifying weaknesses in 


“i Qapac Supa - Did anyone have motive, the previous analysis is encouraged, the group 


contribution to improving the analysis. 


" Absenté of Information — Were there major 
information gaps? 


Common 1 Qali Pitfalls = Were alternative 
hypotheses considered? “Woulda change in the | Weaknesses 
broad environment (technological change, 
globalisation, environmental change, éte,) have | Egos and lack of imagination may interfere. 


had an impact on the analysis? 
Source: Handbook of Analytic Tools & Techniques, 
Pherson, 2008. 
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Structured Analytic Technique 


Quality of information Check 
Purpose 

To provide a firm foundation for analysis. 
Description 


Information is the raw material of analysis. Data, 
facts, opinions, evidence, and proof are all types 
of information. 


Analysts use information from a variety of 
sources. Each source has its pros and cons (see 
INT Comparison Chart). 


By processing information, analysts transform it 
into one of four basic types of intelligence: 
Basic, Current, Estimative, or Warnings Thefirst 
step in this transformation is usually coalition, 
organizing raw information. The second is 
evaluation. 


Usage 
Evaluating the quality of information is crucial-to 
estimating probability and expressing a level of 


confidence concerning a judgement. 


A quality of information check may uncover 
enemy attempts at denial and deception (D&D). 


Analysts should periodically reconfirm the 
information on which their analytic judgments 
are based. 


Requirements 


;,| To evaluate quality, analysts need to know: 
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de, Who or what providedithe information? 
“Why? What was;the motive? 
What was the basis 6f their knowledge? 
How was the information collected? Under 
what circumstances? 4 
How was it reported? 


The basic process has seven steps. 


«  Hoów.old is it? When, was the information 
obtained? When was it reported? Is the lag 
a problem? E 


Where did it come from? Which, agency or 
organization? Whats their reputation? 


Where did they get it? What was the origin? 
How many links were in the “chain of 
acquisition”? 


If HUMINT, how are sources described and 
rated? Read the source context statement. 


e How did this source come to know this 
information? Is this plausible? 

e Does the source have the requisite background 
or access to report reliability on this issue? 

e Is the source biased? 


Is the information consistent with previous 
information? 


Does the information make sense? Is it 
plausible? Are there any errors of fact? 


Could this be a deception? Is the 
information too good or too specific? 


Strengths 


Having multiple sources on an issue is not a 
substitute for having good information that has 
been thoroughly examined. 


Weaknesses 


If no objective criteria or basis of comparison 
exists, it is difficult to say that one thing is better 
than another. 


For reasons of security, it is often difficult to 
know the full context and conditions under 
which critical information has been collected. 


Errors in processing, translation, or 
interpretation may garble the message. 
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UNCLASSIFIED 


Structured Analytic, Technique 


Red Cell Requirements 


A Red Cell needs a diverse group of people 
who are-"out of the box” thinkers: Not afraid to 
speak out or appear the fool, they should be 
ingenious and resourceful. 


Purpose 


A Red Cell is an internal think tank created to 
help an organization develop broader peripheral 
vision and avoid surprise. 


Members could include psychologists, 
philosophers, professors, and/analysts from 
various units and agencies who think differently 
from the ones who typically work on the 
account (i.e. file / issue / problem). 


Description 


In response to 911, the CIA created a Red Cell 
that would think unconventionally about the full 
range of relevant analytic issues. Their short 
"think pieces" about plausible events for which 
there was currently little or no evidence were 
intended to provoke thought rather than to 
provide authoritative assessment. The DHS set 
up a similar program in 2004: 


Outside technical experts may be included or 
consulted (depending on the problem). 


To ensure a continual flow of fresh insights, 
new members can be rotated in every three 
months. 


A Red Cell does not automatically-take the 
opposite or contrarian:view —it-is not a challenge, 
technique. It merely'tries to raise awareness. 
The process starts with a question-such as: 
"Could the UN in New York be attacked using 
explosive laden garbage barges?" 


Management should give permanent Red Cell 
units lots of latitude (two thirds of CIA Red Cell 
reports are self-initiated). 


Strengths 


This imaginative technique helps to combat 
assümptions, mindsets, and biases. 


Any feasible question is OK — although, ideally it 
should be one that no one else has asked. The 
next step is generating a series of events or 
scenario that would lead to the event occurring, 
i.e. "What if?" analysis. Finally, a short but 
(hopefully) convincing argument is presented 
that summarizes the evidence and reasoning. 


Weaknesses 


A Red Cell that constantly critiques or “shows- 
up" the analysis of others can create internal 
tensions. Reports may provoke hostility from 
overtaxed workers. "As if we didn't have 
enough problems already! 


Usage 


A Red ‘Cell canbe permanent or ad hoc to deal 
with specific issues (in. which case it might be 

. | referred to asa "Team A/Team B” exercise or a 
^'iRed Team" if it simulatesthe actions or 

^ [decision-making process of an adversary). 


A Red Cell that "colours too far outside the 
lines" risks being ignored or mocked. 


If Red Cell reports undermining mainstream 
analysis are distributed externally, the agency 
could appear indecisive. 


The goal ofa Réd,Cellreport is to persuade 
experts, analyst$; and decision makers to re- 
examine their views and:temporarily re-task 
resources to prove or disprove, the theory 
suggested by the Red Cell. : 


Small organizations may not be able to afford 
removing staff from their regular duties to work 
in a Red Cell. 


Some argue creativity cannot be forced. 
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Structured Analytic Technique 


6. Howis the culture changing? What are 
some signs of vibrancy or decay, e.g., 
increasing membership or disillusionment? 


Red Hat (Anthropology) 
Checklist for Understanding Groups 


Purpose i 
P ~ What unusual values, attitudes ‘or beliefs 


are present? What are the taboos? 


To obtain a deeper and better understanding of 
a group, tribe, or organization. 


8. How are outsiders viewed? Is-this group 
xenophobic, security Conscious, etc? 


Description 


9. What are the capabilities and intentions of 
this group? 


Initially, it may seem that a group is 
homogenous. Rarely is this the case. 
10. Do members of this group think as you 
do? 


Depending on the complexity of the group in 
question, it may take several months or years to 
fully understand its dynamics. 


In each case, analysts should be resourceful in 
looking for indicators (of fractionalization, 
power, etc.). 


The Checklist for Understanding Groups. is a 
series of questions to provoke and guide efforts 
to "fill in the holes" more rapidly and more 
thoroughly than would be the:case with a less 
structured approach: 


Requirements 


A degree in anthropology or sociology is useful. 


Usage Strengths 


Analysts begin by asking the following questions. | With, deeper understanding of a group, it 
becomes easier to penetrate, investigate, 
disruptior prosecute; tactics can be developed 
for engagement and manipulation; and 
potential problem areas for misunderstanding 


or misinterpretation can be identified. 


1. What defines this group (geography, history, 
purpose, etc)? 


. ls it homogenous or are there internal 
factions (religious, ethnic, linguistic, 
generational)? 


Weaknesses 


. How is power distributed? Is it matriarchal or 
patriarchal? Is it based on wealth, fear or 
moral authority? Are positions earned or 
inherited? , 


Understanding another culture (or sub-culture) 
is difficult. Often, it is only possible to observe. 


This etic (outsider) perspective is fraught with 
risk. Whatever the observer is allowed to see 
may be orchestrated to mislead. 


4. What motivates, ard inhibits individual 
^... ^ cceniduct,-e.g., shame-or face (prestige; 
“U honor; reputation)? 


It is preferable to understand behaviour from 
the insider's perspective. However, this emic 
(insider) understanding is much harder to get. 


5. What pressures and stresses (internal and 
external) are present'on.the-group? E.G. Are 
they running out of money and. food? Is 
another gang trying to take over their 
territory? 
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Cultural Intelligence in Iraq 


Human Terrain Teams (HTT) composed of 
cultural experts have been deployed by the US 
Army in Iraq. Their mission is to diagram Iraq's 
cultural landscape — its "human terrain" — in the 
same way intelligence analysts map out its 
cities, roads, and rivers. 


David Matsuda, a cultural anthropologist from 
California State University, is attached to the 
HTT for the 82nd Airborne Division's 2nd 
Brigade Combat Team which operates in 
Northeast Baghdad and Sadr City. When US 
soldiers were doubtful that a food distribution 
depot could be kept secure, Matsuda explained 
how the tribal relationships of the depot's Chief 
of Security protected it. 


Matsuda is the social scientist on a team that 
also includes a chief, an area specialist, and a 
research manager. Matsuda is a civilian while 
the other members are active duty army. 


To accomplish its mission, thé team uses 
information that has already been collected and 
information that the team members collect 
themselves. They then analyze the information 
and present their conclusions and advice to the 
brigade commander. 


Outside the military, however, the teams have 
sparked some controversy. Much of the 
opposition has come from people in the 
academic world, who, according to Matsuda, 
fear that the army will misuse the knowledge 
offered by social scientists. 


"(Some aré)/sayingianthropology can't be part 
of the Army-without-being-corrupted,” he said. 


“A Matsuda said some-of the-coricerns are valid, 
E ;and some are, motivatéd by kriee-jérk anti- 
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militarism J ‘Regardless, he said; thé Stakes are 
too high‘in Iraq right now to sit on the-sidelines. 


Anthropologists beliave thát/all societies 
operate according to a certain, script,” Matsuda 
said. Iraqis have one script, Americans have 
another. 


The HTT's mission is to provide an 
interpretation of the-Iragi cultural script that will 
help soldiers. make the right decisions. 


The team has carried out that task in ways both 
small and large. When the brigade was about to 
put out a wanted poster featuring an image of 
the scales of justice Matsuda pointed out that 
the idea of the scales of justice wasa Greek- 
derived, Western concept that, meant nothing to 
Iraqis. Instead he proposed changing the poster 
to show two open hands - an image drawn from 
ideas in the Quran — in order to make it more 
resonant with Iraqis. 


“We try to find the assumptions and motivations 
behind what people do,” Matsuda said. 


Source: Culture Experts: Human Terrain Team helps 
Soldiers in Iraq understand the cultural landscape by Sgt. 
Mike Pryor, 2nd BCT, 82nd Airborne Division, Public 
Affairs. 


Further Reading: 


- Army Enlists Anthropology In War Zones, New York 
Times, Oct 5, 2007. Combat operations reduced by 
60 percent after scientists arrive. 

- The Human Terrain Mapping: A CORDS for the 21“ 
Century, (pdf), Military Review, Sept-Oct 2006. 

- Human Terrain System. Bridging the Gap: Making 
Social Science Operationally Relevant, (ppt), Lt. Col. 
Edward Villacres, Team Leader, HTT, 82nd Airborne 
Division's 2nd Brigade Combat Team, (2007). 

- The Laboratory for Human Terrain, Dartmouth 
University. 


55 


UNCLASSIFIED 


Structured Analyti¢-Techniques 


moo 


Red October 
Purpose 


To understand what an adversary is planning, 
asses the feasibility of their plans, and the 
impact of their actions. 


Description 


Red October (RO) combines Red Teaming with 
feasibility analysis. It can reveal when a terrorist 
group’s ambitions exceed its capabilities or 
when an attack scenario is unworkable. 


In Phase One, the facilitator tells participants 
that they will take on the role of an adversary 
who is planning an attack. Their job is to, identify 
the adversary's most likely targets and their 
tactics, techniques, and procedures (T+ Ps). 


The facilitator begins by; defining the question, 
E.G. “What kind of successful,attack can al- 
Qa'ida in the Arabian Peninsula (AQAP) 
conduct against critical infrastructure in 
Yemen?" 


He then asks the group to describe what they 
(as the adversary) would consider "success". 
How do they see their capabilities? 


The facilitator may guide participants through an 
group discussion (who, what, where, why, when 
and how) using Structured Brainstorming. 


The facilitator captures ideas raised during the 
discussion, thenpfollowing the session, 
summarizes SRY Tag into an "attack plan" . 


; : n;Phidse Two, a Separate group of.technical 
“experts détermine the feasibility-of the attack 
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plan; ‘damage it would cause, and theTregional 
and global effects They discuss the viability of 
the plan and the’ impact’ of a Successful attack. 


In Phase Three (optional). everyone comes 
together. The participants get feedback from the 
experts. Everyone discusses the effects of 
biases, expectations, assumptions, and miror 


Usage 


Analysts have used the, RO methodology to 
determine how a.terrorist group would attack a 
known infrastructure target. - 


Requirements 


Choosing the right group of participants-and 
technical experts is critical. Participants are 
typically regional subject matter experts who 
can address the intent and capability of the 
adversary. Technical experts come from the 
military and emergency management. They 
could include doctors, economists, and 
engineers. 


In advance of Phase One, participants should 
be provided with a briefing package. It should 
describe the adversary (goals, history, 
Strengths, and weaknesses) and provide 
information-on potential targets of an attack. 


Strengths 


Red Téaming does not address the impact and 
consequences of an attack. RO does. 


The RO methodology gives analysts insight into 
actors' thinking about the planning, execution, 
impact, and success of a plan or mission. 


The methodology develops specific signposts 
and indicators to drive collection, a deeper 
understanding of the technical aspects of 
attacking the target, and bounds the implications 
of a targeted attack in more realistic parameters 
for policymakers. 


RO mitigates potential mindsets, anchoring, and 
expertise biases that could narrow or 

overstate their perspective of the threat that 
terrorist groups pose to known targets. 
Weaknesses 


Mirror imaging is difficult to overcome. 


Source: US National Counterterrorism Center, 2014 
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Red Team Requirements 


Members-of a Red Team need to’have a deep 


Purpose 
ndr ans of.the adversary. This includes 


To simulate the thoughts and actions of an 


adversary. cultural norms; values; and motives, 
= operating doctrine and environment, 
Description a sensitivity to risk, and 


a their subjective understanding of our 


In military exercises, the Red Team represents intentions and capabilities. 


the adversary, the Blue Team our own forces. 


An effective Red Team immerses itself in.the 
circumstances of the adversary and reacts to 
stimuli as they would. 


The team asks “first-person” questions like: 


What guides my actions? Am | a risk-taker? 

a What would my peers, family, or tribe expect 
me to do in the face of ...? 

« How do | perceive and react to external 

. "4threats.and opportunities? 

» “>What personal factors influence my decision 

making? Age? Religion? Parenthood? 


The Red Team follows the techniques, tactics, 
and procedures (TTPs)- and uses the same 
equipment as the adversary. It tries to emulate 
all their capabilities and weaknesses: : 


The team takes full advantage of the "native" 


The idea is to test Red against Blue, exposing" ^ knowlédge of its members, irrespective of rank. 


vulnerabilities and weaknesses. This role- 
playing conflict can be real, simulated, or 


Í : Strengths 
imagined. 


A Red Team frees itself of the mind-set of the 
Blue Team. This simple shift in perspective may 
lead to new insights into the courses of action 

available and acceptable to an adversary. 


The approach can be extended to any scenario 
involving an adversary. For example, to test its 
computer security, an organization could ask its 
own IT department to form a “Tiger Team” 
which would then try to hack into its systems. 


A Red Team can overcome the tendency to 
assume others think and perceive the world as 
we do (mirror imaging). It may explain the 

seemingly irrational acts of a foreign leader. 


The procis, often: starts with a simple question, 
E.G. "If | Were à money Jaunderer, how would | 
AGY ta get around the. rules, and regulations?” 

C Weaknesses 
KUN vatalyst thin sanctions into ani “actor” 
operating within, the adversary s culture and 


fica mih An adversary without doctrine is difficult to 
political milieu: 


imitate. 
Usage Testing oneself against a Red Team that is 


I , TONA s. artificially constrained, is not a real test. 
An analytic Red Team exercise can help, predict 


the actions and reactions of an adversary, both 


: ; This technique requires a lot of time. 
at the tactical and strategic levels. 
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How to write a "Red Team" Paper 


There are three approaches to writing "Red 
Team" papers. 


e Standard 
e Native Guide 
e Fellow Traveler 


All three approaches begin by examining the 
adversary's values, options, and constraints to 
develop a theoretical model of their thoughts 
and potential actions. 


This is called "Deductive Red Teaming." The 
goal is to predict how the adversary would 
respond to a particular stimulus. 


Standard Red Team Paper 


A "Standard" red team paper hasa detached; 
3rd party narrative. E.G. "If the United: States. 
were to do X, given past behaviour and current 
constraints, Russia could/do Y^ 


This can be given more impact by including 
actual quotes. E.G. On April 27th, in a speech. 
to Russian businessmen, Putin assured them 
that the government will do "whatever is 


necessary" to protect business interests abroad. 


Native Guide Red Team Paper 


A "Native Guide" paper presents the 

issue from the perspective of someone “on the 
ground", i.e. someone directly affected by 
circumstances. It could be the view of a crisis 
through the eyes of a foreign leader. This helps 
the readér/appréciate their world view (and their 
depth of hatred Or ESPN). 


` The technique was Used | in "939 by R.V. Jones 
Who. was. having trouble convincing:his 
superiours about Gétman technical 
advancements, Jn two,of his reports to the 
British Air Ministry /he-included imaginary 
conversations between Germán officers in his 
reports to make his reports more convincing. 
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Writing on the state of the Afghan army and 
worried that your paper could end up being 
rather dry and boring with: lots of statistics? Try 
the Native Guide approach! 


How? What about a, paper written from the 
perspective.of à^Day in the Life" of an Afghan 
army private?" It.coutd.cover/all the same issues 
as a regular paper (morale, training,;equipment, 
etc.), but from the point of view of a soldier. 


Paint a vivid enough picture and readers cannot 
help but imagine themselves in the 
circumstances of the protagonist. 


The Native Guide approach is used all the time 
by journalists. Reporting on a famine or flood, 
they'll personalize the story by talking to a victim 
or an aid worker. This makes it easier for 
readers to relate to the story. It also makes the 
story more visceral and more memorable. 


^The Native Guide approach works best with 


readers who like scenarios. However, it does 
not work well with all readers (some see it as 
going “overthe top") and it can be over-used. 


Fellow Traveler Red Team Paper 


The last approach for a Red Team paper is 
"Fellow Traveler". It is recommended for 
situations where you have no information on the 
adversary but are still required to write a paper. 


The best you can do in such circumstances, is 
reasoning by way of historical analogy. 


Don't know anything about insurgents in 
Afghanistan? Write about insurgencies (in 
general) drawing on the Basques, 

Chechneyans and Tuaregs. Reach into history if 
need be. 


While a "Fellow Traveler" paper is highly 
theoretical, it still provides a basis for 
discussion. 


Footnotes: 


1. Jones, R.V. Most Secret War: British Scientific 
Intelligence 1939-1945. Penguin Press, 1978, 2009. 
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Structured Analytic Techniques 


Rumsfeld Square The following order is recommended. 


Purpose e Known knowns: What do we already know? 
e Known unknowns: What./do we need to find 


To force a more comprehensive consideration out? 
of a problem and identify areas of risk. Unknown unknowns; Are there 
issues/aspects that wë are not’considering? 
Description Unknown knowns: Could-the answers 
already be somewhere in oúr databases? 
On Feb. 12, 2002 US Secretary of Defense ¿ 
Donald Rumsfeld explained to the press that: By the end, each square should be filled with a 
set of facts, questions, and suggestions for 
There are known knowns; there are things follow-up. 
we know we know. We also know there 
are known unknowns; that is to say we Strengths 
know there are some things we do not 
know. But there are also unknown The Rumsfeld Square is a meta-planning tool 
unknowns - there are things we do not that encompasses several approaches and 
know we don't know. uses multiple structured analytical techniques 
(SATs). 
While “known unknowns*.are identified 
intelligence gaps, Rumsfeld's. concern, was op Known knowns 
being surprised by an unsuspected problem or ^e, Mind map / Concept map 
issue, an “unknown unknown," things. that<a e Quality of information check 


commander doesn't even know he doesn't ;,e Data visualization (charts, timelines, etc.) 
know Knowneunknowns 


e Antelligence Collection Plan 


V x z " e/~-Collaboration/Outreach 
Rumsfeld didn't mention the “unknown knowns, UnknowA^ankH vns 


that is, the things we don't realize we know. The e  AltemáiW Futures 
complete set can be captured in a “Rumsfeld e Cone of Plausibility 
Square”. e Red Cell 
e  Collaboration/Outreach 
“Known “Known Unknown knowns 
Knowns" Unknowns" e Data mining 
“Unknown “Unknown e Knowledge management 


Knowns" Unknowns" — 
It is common to focus only on finding answers to 


the “known unknowns’. Using the Rumsfeld 
Square, analysts may find reliability issues with 
their “known knowns’, they may re-interpret the 
problem after discovering an “unknown 
unknown” or they may save time and money by 
uncovering and exploiting an “unknown known’. 


Usage 


|The Rümsfeld Square cari be very helpful for 
- | planning a research projector improving a 
collection plan... 


Requirements — 
Ideally, a team is assembled to consider the 

problem or task at hand. Systematically, they 
work their way through each quadrant of the 

Rumsfeld Square, listing items as they go. 


This technique works best as a group exercise 
facilitated by an analytic methodologist. 
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Structured Analytic: Techniques 


Social Network Analysis 
Purpose 


To reveal patterns of interaction and the 
allocation of social and personal capital within a 
group of people. 


Description 


Social Network Analysis (SNA) uses 
sociograms and mathematical computations to 
reveal significant aspects of behaviour in 
groups. The sociograms (network analysis 
charts) can show the relative importance of 
individuals in a network, and the nature and 
relative strengths of the links between 
individuals. 


Concepts used in SNA include: 


a Centrality - the relative position of an 
individual within a network. , Centrality reveals 
influence and control over communications’ . 
and other exchanges. ; 


Density - the number of links in a network a$ ° 


a percentage of the number of possible links. 
In general, high density groups adapt more 
easily to the loss of an individual, while low 
density groups are harmed less by an 
informant in their midst. 


= Redundancy - the extent to which two or 
more individuals are equivalent to each other. 
The equivalence may be in terms of their 
network role, or their functional capabilities. 


a Cutpóihts - - individuals who are the sole 
connections between-network components. 


d Usage ^ 
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SNÁI [r3 emerging asa useful technique in 
targeting intelligence: Itan break groups into: 
smaller, separaté entities; and can identify 
individuals whose removal woüld; cause the 
greatest disruption in the groüp's functioning. 


e uoi. 
SNA is an evolving interdisciplinary technique 


with diverse approaches from Anthropology, 
Mathematics, Psychology, and Sociology. 


Requirements 


Detailed information about the connections 
between individuals in the,group-under study, 
including the nature and content of thog, 
connections. 


Strengths 


Identifies the structural strengths and 
vulnerabilities of a group. In groups with a 
hierarchical structure, SNA reveals patterns of 
communication, exchange and influence. In 
groups without formal hierarchy or structure, the 
social network is the primary organizing 
principal. 


Weaknesses 


l Requités rich/data about the content of links 


between individuals, which can be difficult to 
acquire when studying covert groups. 


References 


s Social Network’Analysis as an Approach to Combat 
Terrorism: Past, Present, and Future Research Steve 
Ressler, Homeland Security Affairs, Vol. Il, No. 2 (July 
2006). 


A Social Network Analysis of Aum Shinrikyo: 


Understanding Terrorism in Australia — Stuart 
Koschade, Ph.D. Candidate Centre for Social Change 


Research Queensland University of Technology, 
October 28, 2005. 

Understanding Jihadi Networks, Strategic Insights, 
Volume IV, Issue 4 (April 2005) by Marc Sageman, 
M.D., Ph.D. 


=" Complex Social and Value Networks: The Jihadi Case 
(Global Network Terrorism (ppt) — A database of over 
500 Southeast Asian Jihadis. 
*Notions of Position in Social Network Analysis" pp 1- 
35 in Sociological Methodology, Vol 22. Stephen P. 
Borgatti and Martin G. Everett, 1992. 
“Destabilizing Networks" pp 79-92 in Connections 
24(3). Kathleen M. Carley, Ju-Sung Lee, and David 
Krackhardt, 2002. 
*The Network Paradigm Applied to Criminal 
Organizations", Peter Klerks, 2001. 


SNA Measures of Centrality 


Degree Centrality 
Degree = 5 


Definition Node with the most 
connections 


Interpretation "Knows a lot" or 
"Well known" 


Closeness Centrality 


Definition Node closest to all 
other nodes 


Interpretation Fast access to 
information 


High Level Leaders 


Definition High betweenness, 
high eigenvector 


Interpretation Executive 
Officer of a company, Military 
Commander 


Gatekeepers 


D~ Definition Low total degree, 
2 ^ high eigenvector 

"Se. Interpretation Executive 
Oassistant, courier 
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.Betweenness Centrality 


"Definition Node in the best path 
/-1o get to everyone else 


Interpretation "Collaborator" or 
"Group Connector ' 


Eigenvector Centrality 


Definition Node most connected 
to other most connected nodes 


Interpretation "Snobbishness" 
or "Strong Social Capital" 


Emerging Leaders 


Definition High total degree, low 
eigenvector 


Interpretation Mid-level 
executive in a company, cell- 
leader in a terrorist organization 


Boundary Spanners 
(potential) 


Definition Low total degree, 
high betweenness 


Interpretation Facilitators, go- 
betweens, fianciers 


= In2008,a idee. student Abh Mercyhust tege uséd:SNA to better understand terrorist enlistment. Her report 
dr ot Gom y S a fpes job of laying out the strengths and weaknesses of SNA in attribute analysis. 


e A Social Network Analysis of Jemaah: ISlamiyah: The Applications to Counterterrorism and Intelligence Stuart 
Koschade, School of Humanities & Human Services, Queensland University of Technology, Carseldine, Australia Studies in 
Conflict and Terrorism, Volume 29, Number 6 September 2006, Pages: 559 — 575. This article describes a social network 
analysis of the Jemaah Islamiyah cell that was responsible for the Bali bombings in 2002. 
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Structured Analytic Technique 


Structured Comparison / 
Multiple Attribute Analysis 


Purpose 


To rank-order priorities for complex issues with 
large sets of options. 


Description 


This technique uses a set of defined criteria to 
allow a formal comparison of entities or 
phenomena. 


The set of attributes (criteria, qualities) and their 
values used will be specific to each problem. 


The set of attributes establishes a common 
ground for comparison among the range of 
options. 


Usage 


Multiple Attribute Analysis / Multiple Criteria 
Analysis are approaches to decision making 
used in the private sector, military and 
government. 


In the context of intelligence analysis, structured 
comparison tools have to be specific to the 
decision-making problem. For example, an 
analytic tool designed to rank-order organized 
crime groups in terms of their strengths and 
weaknesses will not be valid for terrorist groups. 


Requirements 


The attribute setmust be developed by the 
appropriate range of subject matter experts in 
pil order to ensure.relevance to the problem. 


T The arite must have Clear working definitions 
to calibrate. the technique, and so ensure the 
reliability of the FORMA: 


Analysts will ree to have a good: 
understanding of the criteria as defined in the 
technique in order to apply it. 


© gi OLMO 


Strengths 


More objective, comprehensive, and reliable 
than, unstructured. comparison exercises. 


Can be repeated over time, and can be applied 
by multiple groups of analysts. 


Increases credibility of rank ordering; and 
permits auditing of the process. 


Organized Crime in Canada 
Rank Ordered Matrix of 13 Groups 


. Nineteen Evaluative Criteria 


Weaknesses 


Time and resources are needed to create 
specific attribute sets. 


If the process and results are not reviewed, 
there is the potential for respondents to slant 
their input, consciously or unconsciously, in an 
attempt to promote their preferred results. 


Recommended Readings 


Multiple attribute analysis is taught in business 
management, so there is detailed information 
available on the Internet. 


= Project Sleipnir — An Analytical Technique for 


Operational Priority Setting, a paper presented by 
Steven J. Strang, RCMP, at the 2005 International 


Conference on Intelligence Analysis, McLean, VA,.May 
2 to 4, 2005. 


Team A/TeamB 
Purpose 


To provide a self-check that confirms, refutes, or 
offers an alternative (competing) assessment. 


Description 


Team A / Team B is a competition, akin to a 
debate. However, unlike Devil’s Advocacy, 
which simply tries to refute an argument, it 
permits the expression of different views. 


Separate teams are given the same problem. 
They may be told to take opposite sides or left 
to arrive at their own independent conclusions. 


Usage 


When debate is polarized between two views, 
the Team A / Team B;approach helps clarify the 
differences between the two,viewpoints. 


When analysts have developed an inflexible 
mindset, it can force a re-evaluation. 


Requirements 
In the Analysis Phase: 


- Begin by identifying the two (or more) 
competing hypotheses or points of view. 

- Form teams (or designate individuals) to 
develop the best case that can be made for 
each hypothesis. 


In the Debate Phase: 


^c. Both teams present their arguments before 

— an independent jury o ^ 

— The teams Critique each other's 
assessments then respond to Criticisms; 


The jury listens, questions, ánd decides which 
team has the stronger case“It may also 
recommend actions to be taken, e.g. further 
research. 
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Strengths 


Assigning analysts to the team that fits their 
existing view expedites the process.and makes 
for strong arguments: The reverse takes more 
time, but forcing analysts to-argue “the other 
side" can often make them more aware of their 
own assumptions, mindsets and biases. 


The process can help opposing experts, see the 
merit in each other’s arguments. It also allows 
those with opposing views the opportunity to 
express their views and feel that they have been 
heard. 


For the decision-maker, this technique exposes 
and explains important analytic differences 
within the expert community. Senior officials 
often learn more by weighing well-argued 
conflicting views than from reading an 
assessment that masks substantive differences 
or drives analysts to the lowest common 
denominator. 


Weaknesses 


The process. requires time and resources. A 
small organization covering a large area may 
not be able to afford temporarily reallocating 
several analysts to work on the same problem. 


Competition may lead to workplace acrimony. 


Results may be skewed if one team has a 
political agenda (E.G. CIA Team B experiment). 


Further Reading 


= SourceWatch and Wikipedia (user-contributed articles). 

= Team B Intelligence Coups (pdf) by Gordon R. 
Mitchell, Quarterly Journal of Speech, Vol. 92, No. 2, 
May 2006, pp. 144-173. 

= it's Time to Bench "Team B" by L. Korb, Aug 2004. 

= Team B: The Reality Behind the Myth, Richard Pipes, 
Commentary Magazine, Oct 1986. Subscribers only. 

= “Re-examining the Team A-Team B Exercise" by 
Robert C. Reich, International Journal of Intelligence 
and Counterlntelligence, Volume 3, Issue 3 1989, pp. 
387-403. 


= Team B: The Trillion Dollar Experiment, Anne Hessing 
Cohn, Bulletin of the Atomic Scientists, April 1993. 


Venn Analysis 


Purpose 
To graphically explore the logic of arguments. 
Description 


Venn diagrams which are commonly used to 
teach set theory in mathematics can also be 
used to illustrate simple set relationships in 
analytic arguments. 


A simple Venn diagram typically shows the 
overlap between two sets. 


In one set, for 

example, are people 
who like pizza. In the 
other set are people 
who like beer. In the 
middle are people 

who like pizza and beer. 


Circles might also be "nested" within:one 
another, showing that one thing is a subset of ` 
another in a hierarchy or taxonomy such as 
kingdom, phylum, class, order, family, genus, 
and species. 


Applied to an argument, Venn diagrams can 
reveal invalid reasoning such as: 


= Cats are mammals. 

= Dogs are 
mammals. 

= Therefore, dogs 
are Cats... x 


Mammals 


p», As the diac shows; dogs are not a subset of 
cats. nor are there any dogs;whichvare also cats. 


| Consider á different argument 


« Cats are mammals), 
= Tigers are cats. 
= Therefore, tigers 

are mammals. 


Mammals 


The Venn diagram shows the argument is true. 
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Usage 


This approach can be used anytime an 
argument involves a portion sof something, e.g. 
the portion of total foreign investment in Canada 
from “Freedonia”. It is also useful when the 
subject at hand néeds to'meet a.narrow set of 
conditions, e.g. the “window of Spgptunity" fora 
space launch. 


Requirements 


Venn analysis can be done individually, but 
works very well when done in groups. Here are 
the steps: 


« Represent the elements of a statement or 
argument as circles. Use large circles for 
large concepts or quantities. 


a Examine the boundaries of these circles. Are 
they well defined or "fuzzy"? How are these 
things determined, measured, or counted? 


« Consider the impact of time. Are the circles 
growing or Shrinking? This is especially 
important when looking at trends. 


- Check the relative size and relationships 
between the circles. Is it accurate? Are any 
assumptions being made? 


= Zoom-in / Zoom-out. Are there elements 
within circles? Can larger circles be added? 


= Examine and compare overlapping areas. 
What is found in each one? What is 
significant about their relative sizes? What is 
changing over time? 


Strengths 


Visualizing logical relationships often reveals 
flaws in reasoning and prompts deeper inquiry. 


Weaknesses 


Diagrams can become 
complicated 


Further Reading s 
» Symbolic Logic, Peter Suber, Earlham College 
= Symbolic Logic, Lee Archie, Lander University. 


OOOO... eee 
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Venn Analysis — An Example 
Consider the following statement: 


“The recent investments of state-owned 
Freedonian businesses in Canada’s 
strategic sectors are a risk to national 
security.” 


The flow of funds is: 
Canada 


Freedonia — 


On the Freedonian side, reference is made 
to “state-owned” businesses. 

A circle could be Freedonia 
drawn with several 
smaller circles 
within in it 
representing the 
state-owned 
businesses. If we 
"zoom-out', a larger 
circle, 
encapsulating the 
first, could show 
“All Companies’. 


State-owned 


All campanies 


This allows us to add circles for companies 
that are in the private sector as well as 
some that are partially state-owned. 


This simple diagram raises questions such 
as: 


= What percentage constitutes "state- 
owned"? 

= What constitutes "controlling interest"? 

« In átotalitarian state (like the mythical 
Freedonia), is this distinction even 
meaningiok 


SJ > K also highlights : an assumption: 
| State-owtied = State-controlled 


Other Semiotus molei in the 
statement are: l 


State-owned = Hostile 
Private Sector = OK 
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The statement also mentions “Canada’s strategic 
sectors". If this.is represented by a circle, an 
overlapping circle could show that some investment 
in-these.sectors comes from foreign sources, 
including, Freedonia. 


Is this 


^ | l 
Who else invests in pe growing? 


Canada's strategic 
sectors? 


Total Foreign 
Investment 


[Canada How is it 
— measured? 


j Canada's 
| Strategic 
š sectors 


l 
— Size relative to other countries? 
is it growing relative to other countries? 


a; This 
Venn diagram reveals additional matters that need to 
be addressed. 


First, consider the boundaries of the circles. Do we 
know the amount of foreign investment in Canada 
with precision? What is a “strategic” sector? Does it 
include bio-tech? What about nano-tech? 


Second, are these circles growing or shrinking? Is a 
10%/increase in Freedonian investment suspicious if 


|.totalforeign investment grew by 10%? 


Third, examine the relative size and relationships 
between the Gitrcles.'Is the representation accurate? 
Can we assume that Canadian companies are the 
biggest investors in strategic sectors? What 
constitutes a Canadian company? 


Fourth, *zoom-in" to examine overlapping areas. 
Who else invests in Canada's strategic sectors? 
Does Freedonia invest the most? How are its 
investments different from other countries? 


Many analytic arguments high- 
light differences and trends. But 
before arguing something is 
significant, it needs to be put in 
context. 


Freedonia 


Foreign investment 
in Canada by other 


"^" 2 ; countries 
Examining the relationships 


between the overlapping areas of a Venn diagram 
helps puts things in perspective. 


65 


OOOO... wee 


UNCLASSIFIED 


What if? Analysis 
Purpose 

To challenge existing views. 
Description 


Analysts assume that an event has already 
occurred then explain how it came about. 


Usage 


“What If?” analysis is a contrarian technique. It 
is used to challenge an existing judgement or 
strong mind-set. 


Use this technique when a judgement rests on 
limited information or unproven assumptions: 


The technique focuses attention on the 
mechanics of how something could happen, but 
could also be used to explore consequences 
(another good technique in this respect is High- 
Impact/Low-Probability analysis). 


A “What If?” analysis could be presented as a 
sidebar to conventional wisdom. 


Requirements 
Begin by assuming an event has occurred. 


Express this as a statement. 


Identify,key drivers which would shape or affect 
the event; eg. the death of a nation’s leader. 


Ask how this driver would Have. to change in 
ef order to lead to the event/situation in question. 


Ask Stat bates exist that might signal.a 
change in thissdrivers,, 


Ask why or how this change might take place? 
What could trigger it? 
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Strengths 


By shifting the focus from whether an event 
could occur tó how it may happen, analysts 
allow themselves to suspend judgement about 
the likelihood of the event/arid focus more on 
what developments-—even unlikely ones—might 
enable such an outcome. 


The technique leads to developing indicators or 
signposts for the occurrence of the event. 


Identifying the factors that could cause—or 
alter—an event provides decision makers with 
opportunities to shape the future. 


Weaknesses 


It may be difficult to conceive an alternative 
interpretation or reality. Structured 
brainstorming (divergent/convergent) thinking 
may help: 


The full process-is lengthy. The short form is to 
simply select a triggering event that would affect 


Va ey driver and then work backward. 


- Example - 
Assume Russia, ‘overcomes its difficulties. 


“What if?” Statement: Russia in 2020 i isa 
prosperous liberal democracy 


Drivers: The state of Russia is shaped by: x 
1) capacity of the government to act, 2) the 
legitimacy of the regime, and 9) the economy. 


Indicators: To assess the capacity ofthe ` 
Russian government (driver #1), the following 
could be monitored. d x 


* Quality of leadership/organizational 
capabilities 

Responsiveness to popular demands 
Ability to deliver basic goods and services 
Internal Security Capabilities 
Effectiveness of civil/criminal justice 
systems x 
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Structured Analytic Technique EE 0 


Assign weightings to each factor, then multiply 
the weightings by the scores in each cell to get 
wéighted scores (in red). 


Weighted Rankings / Decision Matrix 
Purpose 


To help assess available alternatives by 
weighting criteria in importance. 


Description 


Use this technique whenever you are choosing 
between competing options, i.e. trying to find 
the best choice. It can also be used to simulate 
the decision making process of an adversary. 


Casualties 
Usage 


Weighted Score 


Begin by listing all the options under Raw Score L 


consideration, E.G. A/ Qaida willattack 
Parliament Hill, Union Station, or.Canada's 
Wonderland. 


Notice that, in our example, the preferred target 
changed from Union Station to Parliament Hill. 


Next, determine the factors that,influence the 
choice of options, E.G. A! Qaida likes,to attack 
targets that are iconic, they like to.damage, 
infrastructure, and they want mass casualties. 


End by!conducting a sanity check of the results 
and réviewing'the impact of the weighted criteria 
on.the final result This review should provide 
the insight needed to present the results in a 


: . : clear and persuasive manner to customers. 
Create a matrix by arraying the options against 


the factors. Score each cell. Give it a 10 if the 
option scores high against the factor, E.G. 
Parliament Hill is very iconic, it gets a 10. 


Requirements 


A good (ideally exhaustive) set of options 
should be considered. 


The factors chosen must be relevant and their 
weightings as accurate as possible. 


> | Strengths 
Infrastructure 
4 ^ Aids decision-making by putting things in order. 


- Vgasualties T Aics Avoids the unsystematic and therefore 


a) ; inconsistent use of criteria to make choices. 
^ Raw Store 

Up IG, Weaknesses 
At this point, all factors are weighted equally. To 
make the process more realistic, determine 
which factors have the most influence. EG, If 
the target must be iconic, give it 5096 of the 
weight of the decision. 


Can be overly simplistic. 


Mirror imaging may be a problem if used to 
emulate the thinking of an adversary. 
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Name: 


SAT Review Quiz - Imagination Techniques 


For each situation choose the technique that would be best to use initially. Explain 


your choice. 
1. Alternative Futures 4. Cone of Plausibility 
2. Mind Maps and Concept Maps 5. Force Field Analysis 
3. Structured Brainstorming 6. What if? Analysis 


Situation “A”: The boss wants some new ideas. 


Analytic technique: 


Situation *B": Time is critical and the boss wants three possible scenarios for the future. You don't 
think the situation will change, bütyyou want to consider the most likely variation as well as a remote 
possibility that could have a high impact. 


Analytic technique: 


Situation “C”: The team needs a way to visualize a complicated problem with many inter-connected 
factors. ; 


Analytic technique: 


Situation “D”: The government of a foreign nation may collapse. How could graphically depict the 
dynamic interplay of the forces for change and against change? 


Analytic technique: 


Situation “E”: You want to work backward from a specific potential event to prepare your mind to 
recognize the signs of a significant change. 


Analytic technique: 


Situation «p? You have been asked to generate a series of future scenarios where the First Nations 
of Canada become iricreasingly radicalized and violent. 


Árialytic technique: | 
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Name: 


SAT Review Quiz - Challenge Techniques 


For each “situation” choose the technique that would be best to use initially. 
Explain your choice. 


1. Devil's Advocacy 5. Structured Self-Critique-(Peer Reylew) 
2. Dialectic Inquiry 6. Key Assumptions Check 

3. Outside-In Thinking 7. High Impact — Low Probability 

4. Structured Debate (Team A — Team B) 8. Pre-Mortem 


Situation “A”: There are two opposing points of view on an issue. To highlight rather than bury the 
differences of opinion, you decide to organize two teams to argue both points of view. 


Analytic technique: 
Situation *B": You designate someone to point out the flaws in your argument. 
Analytic technique: 


Situation “C”: You want a more formal test fhát.will'use,a team to systematically expose 
assumptions, test the evidence, and generate alternative. explanations. 


Analytic technique: 


Situation “D”: You decide to assume that time-has passed and the event you predicted has not 
occurred. You ask yourself, "Where did | go wrong?” 


Analytic technique: 

Situation “E: Before publishing, you decide to get feedback from your peers. 

Analytic technique: 

Situation “F: You've heard that bad assumptions are at the root of most intelligence failures. 
Analytic technique; 


Situation “Gr BAs makers are convinced that an event is unlikely but you feel they should be 
waed that a longshot dóve]ogmppt poses a credible risk 


Analytic, téchniques, 
Situation “H: There may be exogenous (external) variables at play that you had not considered. 


Analytic technique: 
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Name: 


SAT Review Quiz - Diagnostic Techniques 


For each “situation” choose the technique that would be best to use initially. Explain your choice. 


1. Analysis of Competing Hypotheses 5. Weighted Rankings / Decision Matrix 
2. Indicators Validator 6. Red Team 

3. Social Network Analysis 7. Structured Comparison 

4. Red Hat (Anthropology) 8. Decision Trees 


Situation *A": To prepare for an upcoming G8/G20 meeting, you need to form a team that will think 
like anti-globalization protestors to help anticipate their activities. 


Analytic technique: 


Situation *B": Several theories have been advanced about the intentions of a local anarchist group. 
How could you "rule-out" some theories as least consistent with the evidence so far? 


Analytic technique: 


Situation “C”: A fragile coalition of groups has formed to oppose the G8/G20 summit. Which 
technique might suggest ways.of fragmenting this coalition? 


Analytic technique: 


Situation “D”: Several locations are available for surveilling the meetings or a radical group. All offer 
advantages and disadvantages. How could you décide which was best? 


Analytic technique: 


Situation “E”: With very limited resources, you can only monitor two of five potential indicators. How 
might you select the best two? 


Analytic technique: 


Situation “F”: What academic discipline would help you to understand a radical environmental 
group? 


Analytic teinu <Q, 

Situation “G”: You want to visualize. possible permutations like the branches of a tree. 

Andivtic technique: 

Situation “H The Boss wants you to prioritize the terrorist groups in your jurisdiction according to a 
given set of criteria. © 


Analytic technique: 
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Glossary of Intelligence Terms 


Abduction: reasoning to the best possible conclusion in light of available-evidence. It is sometimes 
known as intuition, probabilistic assessment, or creative thinking. This type of reasoning is very 
common in intelligence analysis. 

All-source intelligence: the combination of open and clandestine.sources of information. 

Analysis: See Intelligence Analysis. 

Analysis of Competing Hypotheses (ACH): a structured analytic technique that placés;evidence in 
rows and hypotheses in columns. Each piece of evidence is scored against each hypotheses as 
either consistent, inconsistent, or neutral. The technique finds the hypotheses with the best fit to the 
available evidence and disconfirms competing hypotheses. 

Argument: a series of statements (premises) leading to a conclusion. A premise is a supporting 
piece of information (fact, opinion, observation). A conclusion is the end product of an argument; it is 
also known as an inference. 

Assessment: new knowledge by means Such as interpreting data or making a forecast. It is focused 
more on strategic, or long-term, issues. Some intelligence services refer to such assessments as 
estimates. 


Assumption: something which i$'taken'for granted: May be explicit (clearly stated) or implicit (not 
stated). 


Best Practices: processes, procedures, or habits that lead,to good results. 
Caveat: a formal notification or warning. 


Classified Information: information that could reasonably be expected to cause injury to the national 
interest. 


Cognitive Bias — a tendency to think a certain way due to inherent functional limitations of the human 
brain. 


Collation — The process of logging, sorting, and categorizing raw information. A step in the 
intelligence cycle that follows collection and precedes analysis. 


Collection: gatheting data. 


Collection Discipline: the various means by which intelligence (“INT”) is gathered. Includes 
COMINT ELINT; GEOINT, HUMINT, IMINT, OSINT, MASINT, and SIGINT. 


Collection Plán: seé Intelligence Collection Plan. 
Collection Requirements: information needed to satisfy an Intelligence Requirement. 


Communications Intelligence (COMINT); intelligence derived from electromagnetic 
communications and communications systems: A subcategory of SIGINT. 
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Data: information that has not been evaluated or put into context. 


Deduction: A type of reasoning where if the premises are true, then the conclusion is generally true. 
This process is used whenever evidence is offered in support of a hypothesis. 


Denial and Deception: Denial is restricting access to information-about plans, assets; or other ` 
important information, and it is carried out through means such as,silence or concealment. Deception 
is steering observers away from vital information, and it is conducted by providing alternative and 
false information that is still plausible. | 

Devil's Advocacy: a structured analytic technique in which counterarguments are presented: 


Dissemination: the process of distributing analytic products to those who need it in a timely fashion. 


Electronic Intelligence (ELINT): intelligence derived from electromagnetic non-communications 
transmissions. A subcategory of SIGINT. 


Established Source: a Human Source who has been successfully validated and whose relationship 
with the handler is fully developed in terms of direction and control. (Source établie) 


Evidence (Indicators/factors): observable phenomenon or information. 


Facts: information which is judged to be true. Facts are used to substantiate statements and 
arguments. 


Fallacy: an error in reasoning. Logical fallacies: have a flaw in their structure. Informal fallacies have a 
valid structure, but one or more premises are false: 


Feedback: comments on analytic products from readers: 

Foreign Instrumentation Signals Intelligence (FISINT): technicalinformation or intelligence derived 
from the collection, processing and analysis of foreign instrumentation signals such as telemetry, 
beaconry, electronic interrogators, and video data links (drone images transmission). A subcategory 
of SIGINT. 

Geospatial Intelligence (GEOINT): see Imagery Intelligence. 

Human Source: a person who provides information. Their identity is usually protected. 

Human Intelligerice (HUMINT): information provided by human sources. 

Hypotheses: a theory; a/possible explanation. 

Induction: A typeof reasoning where orie-observes a series of events (empirical data) to build a 
theory (general principal), Any generalization is án example of induction. This process is used by 


academics, scientists and strategic analysts. 


Inference: a conclusion that comes from evidence. 
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UNCLASSIFIED 


Imagery intelligence (IMINT): images from traditional hand-held photography, as well as satellites 
and high altitude aircraft. See: Geospatial intelligence (GEOINT). 


Information: unevaluated and unprocessed data of any type that has been openly or clandestinely 
collected. Information when processed may produce intelligence. 


Intelligence: a broad term, used to describe both the input and output of the intelligence process. 
The term could refer to the raw information collected by an intelligence agency:or information that has 
been processed into an analytic product. 


Intelligence Analysis: the process of adding value to information. It involves gathering; organizing, 
evaluating, analyzing, and presenting finished intelligence to decision makers. 


Intelligence Collection Plan: a plan for gathering information from available sources to meet an 
intelligence requirement. 


Intelligence Cycle (or Process): the sequence of activities whereby information is obtained, 
assembled, converted into intelligence and made available to users. This sequence comprises the 
following phases: 


direction: the determination-of intelligence requirements; 
planning: planning the collection effort, and:issuance of orders and requests to collection 
agencies; 

e collection: the exploitation of sources of information and the delivery of the information 
obtained to the appropriate processing unit; 

e analysis: the processing of information into intelligence through collation, evaluation, 
integration and interpretation; : 

e dissemination: the timely conveyance of.intelligence; in an appropriate form and by any 
suitable means, to those who need it. This lastsphase:can include feedback from the client 
which is then used to adapt existing, or formulate néw intelligence requirements. 


Intelligence Requirement - IR: any subject, general or specific, upon which there is a need for the 
collection of information or the production of intelligence. 


Knowledge (or understanding): facts that have meaning because they have been put into a context, 
such as patterns and relationships that provide insight into causes, reasons, and implications. 


Link Diagrams: drawings showing relationships between entities (persons, places, and things). 
These diagrams aid investigators and analysts by uncovering, interpreting, and displaying complex 
information iri/easily-ünderstood chart form. 


Measurement and Signature Intelligence (MASINT): intelligence obtained by quantitative and 
qualitative analysis of eleétromagnetic and acoustic emanations. Includes analysis of physical 
evidérice (E.G; DNA, fingerprints; footprints, fiber samples, smoke from factories, etc). 


Mindset - a set of assumptions about the way the world works. A frame of mind. 


Need-to-know: the principle whereby an individual is provided with access only to the classified or 
designated information needed to properly carry out their current duties or responsibilities. 
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Open Source Intelligence (OSINT) is intelligence based on information collected from sources open 
to the public such as the media, books, or the Internet. 


Processing: putting the data into forms that analysts can use, for example by decrypting or 
translating it. 


Reasoning: the act of forming conclusions. There are three types: inductive, deductive, and) 
abductive. 


Red Team: a structured analytic technique to look at situations from the opponent’s-point of view. 
Risk Assessment: the likelihood of a threat materializing and its consequences. Typically examines 
the probability of the threat, the vulnerability of the target and the impact of an attack or incident to 


determine a risk score. 


Qualitative Analysis: the examination of non-measurable data related to attributes or characteristics 
(subjective properties). 


Quantitative Analysis: the examination of measurable and verifiable data (objective properties). 
Signals Intelligence (SIGINT): interceptéd;signals and audio surveillance. Includes communications 
intelligence (COMINT), electronic intelligence (EEINT) and foreign instrumentation signals intelligence 
(FISINT). 


Structured Analytic Techniques: approaches to analysis that are formal and systematic. 


Tactical Intelligence: Tactical issues involve actual-or imminent combat, and take place in a limited 
timeframe (hours and days) and space (the battlefield). 


Team A/Team B Exercise: a structured analytic technique-whereby two groups are given the same 
data, but they evaluate it separately to see if they come to similar; or different, conclusions. 


Target: an individual, group of persons or organization whose activities are reasonably suspected of 
constituting a threat. 


Threat: any potential event or act, deliberate or accidental, that could cause injury. 


Threat Assessment: the process of identifying threats to things or from things. The “agent of threat” 
may be man-made (e.g. terrorist) or natural (e.g. flood). If man-made, the assessment normally 
covers the motivation, intent and capability of the “agent of threat". A threat "scenario" typically 
identifies the agerit;the-áction, and the target. The threat assessment usually estimates the 
probability of thé scenario: 


Terroristi An individual who ‘has, or will erigage in, assist, commit or conduct a politically, religiously 
or ideologically motivated act of serious violence against persons or property. 


Tradecraft: the set of practices thabhave evolved over time and define a particular activity or trade. 


Operational Intelligence: deals with a-series of battles, or a campaign, that take place over a longer 
period of time (weeks and months) and in.a wider space (theatre of operations). 
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Strategic Intelligence: long term trends, as well as vaigus, ae slg scenarios to better understand 
options, outcomes, and expected reactions. 
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‘The National Bestseller — 
Second Edition with: 
Anse ta "Een Questions Pe ook Ask 


GETTING TO 
MAYBE 


Intelligence Analysis 
Made Easy 
Dewey, Hedge & Howe 
of the Delphi Project 


C] Single 

C] Married 

E]. It's complicated 

{In a Relationship 
Still analyzing 
the question 


UR SAMPLE SIZES ARE SMALL ` 
YOUR STANDARD DEVIATIONS ARE HIGH 
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